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SUMMARY 


This  report  presents  a  methodological  framework  for  analyzing  tradeoffs 
among  alternative  combinations  of  aiding  and  training  for  personnel  in  complex 
systems.  Each  of  fifteen  steps  is  described  in  detail  as  it  applies  to  the 
conceptual  methodology  and  to  its  implementation  as  part  of  a  decision  support 
system  prototype.  Two  operational  analyses  of  AFHRL  approved  specialties 
are  then  presented,  documenting  the  results  of  the  user/system  dialogue  at 
each  step.  Finally,  a  series  of  "lessons  learned"  from  this  phase  of  effort  have 
been  compiled  and  presented  as  they  apply  to  this  and  other  aiding/training 
decision  paradigms. 


I.  INTRODUCTION 


The  ever  increasing  complexity  of  operational  Air  Force  systems  continues  to 
place  greater  demands  on  the  personnel  operating  and  maintaining  them  (AFHRL 
Report,  1986).  The  increased  sophistication  of  these  systems  coupled  with  decreased 
force  levels,  declining  entry-level  skills,  and  the  need  to  limit  military  training  spending 
are  forcing  Manpower,  Personnel,  and  Training  (MPT)  agencies  to  seek  more  efficient 
methods  of  maintaining  and  improving  operational  readiness  (Booher,  1978;  Duncan, 
1985). 

In  this  environment  of  "doing  more  with  less",  the  issues  of  training  and  job  aiding 
are  paramount.  Technical  training  serves  as  the  source  of  knowledge  and  skills 
essential  to  task  performance.  In  other  words,  training  "creates  the  potential  to 
perform"  (Rouse  and  Johnson,  1989).  Job  Aiding,  collectively,  refers  to  those  devices 
with  the  capacity  to  store  and  retrieve  the  "How",  "What",  and  "When"  information 
pertinent  to  a  particular  task  performance.  Job  aiding,  therefore,  directly  augments  the 
potential  to  perform  (Rouse  and  Johnson,  1989). 

Selecting  from  among  the  wide  variety  of  training  and  aiding  alternatives  (and 
their  possible  combinations)  is  difficult  when  a  myriad  of  interdependent  factors  such 
as  performance-related  effectiveness,  development/implementation  costs  and  system 
design  impact  must  be  simultaneously  resolved.  For  example,  as  an  information 
storage  device,  a  job  aid  facilitates  performance  by  reducing  the  task  related  memory 
requirements.  This,  in  turn,  reduces  the  training  requirements  for  that  job  and 
generates  the  potential  for  reducing  immediate  resource  expenditures.  Training,  on 
the  other  hand,  imparts  more  general  knowledge  applicable  to  a  variety  of  related 
tasks.  In  this  case,  the  initial  costs  of  training  a  small,  multi-disciplinary  work  force 
may,  in  the  long-term,  be  offset  the  reduced  costs  of  supporting  a  larger,  more 
specialized  team. 

The  formulation  and  evaluation  of  these  aiding/training  tradeoffs  is  a  necessary 
component  of  the  decisions  made  by  MPT  analysts,  system  designers,  and  personnel 
supervisors  through-out  the  Air  Force.  To  the  extent  that  these  tradeoffs  have  been 
addressed  in  the  past,  the  analyses  have  relied  heavily  on  prior  experience  with 
similar  systems.  Typically,  these  types  of  analyses  have  required  many  person-years 
of  effort.  Often,  the  result  has  been  a  time-consuming  and  expensive  effort  that 
provided  insights  which  were  too  late  to  be  implemented  in  any  substantial  way. 
(Rouse  and  Johnson,  1989)  Whether  for  evaluating  current  AFS  job  performance, 
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selecting  among  new  system  design  alternatives,  or  ensuring  flight-line  personnel  are 
task  qualified,  a  methodology  for  efficiently  producing  consistent,  timely,  and 
supportable  aiding/training  decisions  is  a  must. 

In  response  to  this  need  the  Air  Force  Human  Resources  Laboratory,  Brooks  AFB 
has  sponsored  the  Job  Aiding/Training  Allocation  Technologies  (JATAT)  program. 
The  purpose  of  JATAT  is  to  develop  a  conceptual  decision  aiding  methodology  and  a 
corresponding  computer-based  decision  support  system  designed  to  assist  in 
identifying  applicable  training/aiding  alternatives  and  evaluating  combinations.  The 
expected  benefits  of  such  a  system  include  faster  response  times  to  identified 
aiding/training  requirements,  accurate  performance-based  recommendations,  and 
reduced  military  training  costs. 

This  report  documents  the  second  phase  of  effort  in  the  development  of  the  JATAT 
aiding/training  decision  methodology,  the  Phase  2  prototype  system,  and  the  analyses 
of  two  Air  Force  approved  Air  Force  Specialties  (AFS's)  using  the  JATAT  methodology. 
To  support  this  goal,  this  report  is  divided  into  three  sections;  an  overview  of  each  of 
the  15  steps  in  the  methodology,  the  Phase  2  Decision  Support  System  Prototype, 
and  a  summary  including  "lessons  learned"  during  concept  development  and 
implementation.  Appendix  A  contains  a  step-by-step  analysis  of  the  two  operational 
AFS's. 


II.  METHODOLOGY  OVERVIEW 

Overall  .P..escriptiQii 

In  a  prior  report,  Rouse  and  Johnson  (1989)  suggested  three  computational 
approaches  for  supporting  trade-off  decisions  between  training  and  job  aiding.  The 
first  approach  involves  compiling  general  guidelines  for  training/aiding  decisions 
based  on  cumulative  experience  and  experiments.  This  results  in  a  "rule-based" 
approach  in  which  the  tradeoffs  are  embodied  in  rules  based  on  mappings  from  task 
performance  requirements  to  training/aiding  decisions. 

The  second  approach  involves  predicting  human-machine  system  performance 
based  on  attributes  of  specified  training  and  aiding  alternatives.  This  approach 
enables  the  analyst  to  specify  the  appropriate  performance  measures  and  acceptable 
levels  of  performance  in  different  situations  and  with  different  priorities.  This  requires 
computational  models  that  predict  the  measures  of  interest  based  on  the  available 
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attributes  of  the  training  or  aiding  alternatives.  It  is  a  more  ’visible"  form  of  the  first 
approach  in  that  the  first  approach  may  "hide"  the  measures  and  requisite  levels  of 
performance  in  the  rules  or  guidelines. 

If  the  rules,  guidelines,  or  computational  models  do  not  exist  or  are  not  adequate, 
then  the  analyst  can  use  simulation  techniques  to  estimate  performance  measures  of 
the  human-machine  system  using  different  training  or  aiding  alternatives.  This  third 
approach  is  actually  a  special  case  o*  the  second.  In  this  case,  the  analyst  must 
develop  or  tailor  a  simulation  model  to  predict  the  chosen  measures  of  performance. 

The  methodology  described  in  this  report  encompasses  these  three  approaches 
placing  them  in  the  larger  context  of  a  training/aiding  trade-off  analysis  (see  Table  1). 
Steps  1  and  2  of  the  method  reflect  a  typical  systems  engineering  approach  to  the 
analysis.  Step  3  indicates  a  human-centered  approach  to  identifying  the  requirements 
that  will  be  addressed  in  the  analysis.  Steps  4  through  15  of  the  method  (in  which  the 
analyst  determines  the  alternatives  and  formulates,  analyzes,  and  integrates  the 
tradeoffs)  describe  an  ordered  approach  to  the  complex  problem  of  analyzing  multiple, 
interdependent  tradeoffs  between  aiding  and  training  the  human. 

Although  presented  as  an  ordered  list  in  Table  1,  these  steps  are  not  necessarily 
sequential.  Some  of  the  steps  may  be  repeated  several  times  as  the  analyst/designer 
works  through  the  tradeoffs  under  various  conditions  and  with  various  combinations. 
The  following  sections  discuss  these  steps  in  detail. 

Individual  Step  Description 

1.  IDENTIFY  TASKS  -  Understand  the  Job 

In  the  context  of  analyzing  job  aiding/training  tradeoffs,  the  analyst  must 
understand  three  different  aspects  of  the  job:  the  tasks  involved  in  the  job,  the 
equipment  used  in  the  job,  and  the  personnel  expected  to  do  the  job.  This  knowledge 
is  necessary  to  determine  the  job  requirements  and  system  constraints  that  must  be 
satisfied.  Obviously,  complete  knowledge  of  these  variables  for  all  tasks  is  unrealistic. 
In  fact,  the  required  level  of  understanding  is  directly  dependent  upon  the  problem  at 
hand.  For  example,  a  decision  to  train  or  aid  personnel  to  perform  a  job  requires  far 
less  detailed  information  than  a  decision  among  combinations  of  training  and  aiding 
technologies  for  a  specific  task. 
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Table  1.  Methodology  Overview. 


IDENTIFY  TASKS 

1 .  Understand  the  job 

2.  Decompose  via  a  task  taxonomy 

ASSESS  HUMAN  LIMITATIONS,  ABILITIES,  AND  PREFERENCES 

3.  Assess  human  limitations,  abilities,  and  preferences 

DETERMINE  ALTERNATIVES 

4.  Map  limitations,  abilities,  and  preferences  to  a  taxonomy  of  training  alternative: 

5.  Map  limitations,  abilities,  and  preferences  to  a  taxonomy  of  aiding  alternatives 
FORMULATE  TRADEOFFS 

6.  Make  obvious  choices 

7.  Coalesce  interdependent  tradeoffs 
ANALYZE  TRADEOFFS 

8.  Choose  measures  of  performance 

9.  Choose  input/output  representations 

10.  Identify  requisite  structure  and  parameters  for  representations 

11.  If  necessary,  represent  learning  process 

12.  Apply  methods  of  analysis  to  representations 

13.  Interpret  results 
INTEGRATE  TRADEOFFS 

14.  Compile  assumptions  and  consequences  of  tradeoffs 

1 5.  Form  sets  of  tradeoffs  with  consistent  assumptions  and  consequences 


2.  IDENTIFY  TASKS  -  Decompose  via  Task  Taxonomy 

The  second  step  is  to  further  decompose  the  tasks  into  more  primitive  tasks; 
refered  to  as  subtasks  or  activities.  This  decomposition  defines  the  level  of  granularity 
for  subsequent  steps  (the  assessment  of  the  human's  limitations,  abilities,  and 
preferences).  A  task  taxonomy  is  useful  in  this  step,  particularly  if  the  human’s 
limitations,  abilities,  and  preferences  are  readily  determined  for  the  task  elements  in 
the  taxonomy. 

ASSESS  LIMITATIONS.  ABILITIES.  AND  PREFERENCES 

In  this  step,  the  analyst  determines  the  qualities  of  the  human  in  the  system  that 
either  require  (through  human  limitations)  or  influence  (through  human  abilities  and 
preferences)  training/aiding  decisions.  It  is  this  focus  on  the  human  capabilities, 
limitations,  and  preferences  in  the  system  that  makes  this  a  human-centered 
approach. 

This  assessment  draws  its  primary  input  from  the  task  decomposition  in  the 
previous  step,  which  provides  an  "index"  for  human  limitations,  capabilities,  and 
preferences.  In  subsequent  steps,  these  assessments  will  be  used  to  identify  training 
and  aiding  alternatives.  It  is  important,  then,  to  maintain  a  consistent  level  of  detail 
across  these  tasks,  limitations,  and  alternatives.  A  task  composition  that  is  too  coarse 
leads  to  identifying  general  human  limitations  that  are  not  sensitive  to  the 
aiding/training  alternatives  available.  A  task  decomposition  that  is  too  fine  grained 
leads  to  identifying  human  limitations  that  require  premature  detailed  design  of 
aiding/training  alternatives  to  evaluate. 

1 _ DETERMINE  ALTERNATIVES  -  Map  Limitations.  Abilities,  and  Preferences  to 

Taxonomy  of  Training  Alternatives. 

5, — DETERMINE  ALTERNATIVES  -  Map  Limitations.  Abilities,  and  Preferences  to 
Taxonomy  of  Aiding  Alternatives. 

Through  Steps  4  and  5,  the  analyst  uses  the  limitations  and  abilities  identified  in 
Step  3  to  guide  the  identification  and  selection  of  alternative  training  and  aiding 
techniques.  This  is  done  by  identifying  the  knowledge  and  skill  requirements  of  a  task 
and  mapping  the  required  changes  in  knowledge  and  skills  to  candidate 
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aiding/training  methods  through  guidelines.  The  mapping  is  guided  by  available 
expert  guidelines  or  empirically  developed  heuristics. 

From  a  pragmatic  perspective,  other  factors  may  also  go  into  this  process,  such  as 
resource  availability  and  existing  training  or  aiding  techniques  for  this  or  similar  jobs. 
Depending  on  the  maturity  of  the  analysis  and  the  expertise  of  the  analyst,  these 
considerations  may  either  prematurely  constrain  the  solution  space  (early  in  the 
analysis)  or  provide  timely  guidance  leading  to  practical  solutions  (later  in  the 
analysis). 

FORMULATE  TRADEOFFS  -  Make  Obvious.  Choices 

In  this  step,  the  analyst  selects  among  training/aiding  alternatives  that  are  straight¬ 
forward  and  require  no  additional  analysis.  This  step  allows  for  the  situation  in  which 
part  of  the  problem  is  easily  addressed  by  conventional  solutions.  For  example, 
printed  procedural  job  aids  may  be  an  obvious  solution  for  a  task  which  is  similar  to 
one  already  using  that  type  of  aid  extensively. 

Clearly,  these  choices  depend  upon  the  expertise  of  the  analyst  as  well  as  the 
data  and  tools  available  to  the  analyst.  A  relatively  novice  analyst  may  be  unable  to 
independently  make  obvious  choices,  but  may  be  able  to  rely  upon  tools  such  as 
heuristic  guidelines,  decision  flow  charts  (Booher,  1978),  or  expert  judgement  models 
(Irvin,  Blunt,  &  Lamb,  1988)  for  making  broad  categorical  decisions  (e.g.,  train,  aid, 
both,  or  either).  A  more  experienced  analyst  may  want  to  use  these  tools  to  verify  their 
choices. 

Making  the  obvious  training/aiding  choices  now,  however,  does  not  remove  them 
from  further  consideration.  Their  interdependencies  must  still  be  considered  in  later 
steps. 

7^-FQRMULAIE  TRADEOFFS  -  Coalesce  Tradeoffs  Involving  Interdependent  Tasks 

To  this  point  in  the  analysis,  the  number  of  viable  alternatives  has  been  relatively 
unlimited.  However,  once  obvious  choices  have  been  made,  subsequent  analyses 
can  be  quite  extensive.  Therefore  it  is  usually  necessary  to  narrow  down  the  number 
of  candidate  solutions  by  grouping  training  and  aiding  alternatives  according  to  their 
interdependent  relationships  and  characteristics. 

For  example,  a  particular  task  element  may  suffer  from  a  limitation  that  may  be 
addressed  by  one  of  three  alternatives:  training  alone,  aiding  alone,  or  some 
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combination  of  training  and  aiding.  It  is  likely  that  these  task  elements  will  be 
functionally  or  temporally  interrelated.  Similarly  the  training  and  aiding  alternatives 
will  probably  have  interdependent  characteristics  such  as  methods  or  content.  The 
analyst  must  identify  these  interdependencies  and  coalesce  the  training/aiding 
alternatives  into  a  smaller  set  for  subsequent  analysis. 

8.  ANALYZE  TRADEOFFS  -  Choose  Measures  of  Performance 

Accurately  evaluating  the  resultant  training/aiding  alternatives  requires  selecting 
the  appropriate  performance  measures.  These  measures  are  clearly  domain 
dependent.  Cost,  for  example,  is  a  basic  measure  of  performance  common  to  all 
domains;  although  its  importance  will  vary  accordingly.  Other  examples  include  time 
to  perform,  probability  or  number  of  errors,  mean  time  between  failures,  etc. 

The  choice  of  performance  measures  is  also  influenced  by  the  available  modeling 
tools  and  the  modeling  expertise  of  the  analyst.  While  a  more  experienced  analyst 
may  choose  to  tailor  the  available  modeling  tools  or  develop  new  models  to  produce  a 
variety  of  performance  measures,  a  novice  will  probably  have  to  choose  among  "pre¬ 
determined"  models  that  are  readily  available. 

9.  ANALYZE  TRADEOFFS  -  Choose  Inout/Output  Representations 

To  compare  training/aiding  alternatives,  an  input/output  (I/O)  representation  (i.e. ,  a 
model)  must  be  chosen  that  can  produce  the  selected  measures  of  performance.  The 
I/O  representation  must  reflect  realistic  inputs  from  available  data  and  the  desired 
outputs  including  the  performance  measures. 

Once  again,  the  experience  level  of  the  analyst  strongly  influences  the  extent  of 
this  step.  Again,  while  a  more  experienced  analyst  may  be  able  to  adapt  existing 
models  or  develop  new  ones,  for  a  novice  the  choice  of  I/O  representation  may  follow 
directly  from  the  choice  of  a  performance  measure. 

1(L_ ANALYZE  TRADEOFFS  -  Identify  Requisite  Structures  and  Parameters 

Employing  the  chosen  I/O  representation  frequently  requires  modeling  the  human 
as  an  integral  component  of  the  system.  In  doing  so,  it  may  be  necessary  to  determine 
the  structures  and  parameters  that  represent  how  the  human  performs  the  task.  If  the 
analysis  includes  only  aiding  alternatives,  these  requirements  may  be  essentially 
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constant  throughout  the  analysis.  In  analyses  that  include  training  alternatives,  these 
requirements  will  vary  to  simulate  the  impact  of  different  training  alternatives. 

JLL  ANALYZE  TRADEOFFS  -  If  Necessary^Represent  Learning  Process 

In  some  analyses,  the  performance  measures  may  be  sensitive  to  the  human 
process  of  acquiring  knowledge  and  skills.  In  these  cases,  the  learning  process  must 
be  reflected  in  the  model.  This  representation  may  be  as  simple  as  retrieving  data 
from  a  database  or  as  complex  as  employing  learning  curve  or  learning  process 
models. 

12.  ANALYZE  TRADEOFFS  -  Apply  Methods  of  Analvsis  la. Representations 

This  step  invokes  the  targeted  analysis;  input  data  is  supplied,  the  model  is 
exercised,  and  performance  data  is  collected  for  each  of  the  training/aiding 
alternatives  of  interest. 

13.  ANALYZE  TRADEOFFS  -  Interpret  Results 

Next,  data  collected  during  the  previous  step  is  analyzed  and  interpreted  in  the 
context  of  selected  analyses.  This  step  may  be  repeated  several  times  in  conjunction 
with  steps  10  through  12  as  the  analyst  investigates  the  effects  of  various  assumptions 
or  the  sensitivity  of  the  performance  measures  to  variations  of  the  parameters  within 
the  model. 

14.  INTEGRATE  TRADEOFFS  -  Compile  Assumptions  and  Consequences  of 

Iiadeoils 

In  an  extensive  analysis  with  a  number  of  different  tasks  and  training/aiding 
alternatives,  organizing  the  assumptions  and  consequences  of  the  trade-off  analyses 
is  a  large  bookkeeping  task. 

The  purpose  of  this  step  is  to  compile  all  the  common  aidingAraining  alternative 
characteristics  and  decisions  in  order  to  implement  the  predetermined  aggregation 
guidelines  in  the  following  step. 
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15.  INTEGRATE  TRADEOFFS  -  Form  Sets  of  Tradeoffs  with  Consistent  Assumptions 
and  Consequences 


In  the  final  step  of  the  methodology,  the  training/aiding  alternatives  are  integrated 
into  sets  satisfying  the  requirements  developed  from  the  human  limitations,  abilities, 
and  preferences  identified  in  Step  3.  In  addition  to  satisfying  these  requirements,  each 
set  incorporates  the  common  assumptions  and  consequences  (i.e.,  learning  and 
retention  abilities  of  humans  or  productivity  improvements  with  job  aiding)  identified  in 
the  previous  step. 

While  most  analysts  will  probably  not  have  the  final  decision-making  authority 
necessary  to  implement  the  recommendations  produced  in  a  JATAT  analysis,  this 
methodology  generates  a  logical  justification  supporting  these  recommendations.  The 
purpose  of  this  step,  therefore,  is  to  compile  a  clear,  coherent  summary  of  that 
justification. 


III.  DECISION  SUPPORT  SYSTEM 

Implementing  the  JATAT  decision  support  system  prototype,  as  described  in  the 
following  sections,  presented  several  unique  design  challenges.  First,  unlike  other 
complex  decision  aiding  environments  where  the  end  user  population  is  a  highly 
specialized,  well-defined  segment  of  the  general  populace,  the  characteristics  of  the 
JATAT  user  were  intentionally  left  ambiguous.  This  decision  reflects  our  conclusion 
that  training/aiding  allocation  decisions  are  currently  made  across  a  variety  of 
disciplines,  organizations,  and  individual  experience  levels.  Throughout  our 
preliminary  design  process,  this  ambiguity  allowed  sufficient  design  flexibility  to  satisfy 
numerous  potential  user  functional  needs. 

Second,  a  workable,  yet  intuitive,  user  interface  metaphor  was  needed  to 
adequately  represent  the  JATAT  method  and  to  exploit  a  newly  identified  ’value 
added'  quality  of  the  prototype  system.  This  "value-added"  concept  was  a  result  of  the 
aggregation  of  resources  necessary  to  support  the  decision  process  and  was  a 
natural,  albeit  secondary  product  of  the  implementation  effort.  The  interface  metaphor, 
in  this  case,  graphically  reflects  the  user's  cognitive  model  of  the  system.  More 
specifically,  this  interface  metaphor  design  strives  to  represent  a  conceptually  new 
system  with  an  analogy  familiar  to  the  user  (i.e.,  a  process  control  and  resource 
available  environment). 


9 


Finally,  the  system  complexity  required  the  exploration  ot  an  intelligent  support 
function.  Therefore,  the  Decision  Support  System  (DSS)  section  has  been  further 
divided  into  the  following  four  sub-sections:  the  User,  the  Environments,  the  Advisor, 
and  Support  Data  Types. 


The  User 

An  important  notion  in  the  development  of  a  supportive  system  interface  is  to 
clearly  define  the  user  requirements  and  capabilities.  In  this  case,  an  exact  definition 
of  the  user's  characteristics  was  complicated  by  the  multiplicity  of  intended  JATAT  user 
characteristics.  It  was  hypothesized  that  the  JATAT  "user”  could  fall  anywhere  along  a 
continuum  from  an  individual  tasked  with  making  a  short  response  time,  categorical 
level  aiding/training  recommendation  for  a  previously  established  task,  to  a  team  of 
users  employing  the  JATAT  system  as  an  integral  part  of  a  long-term,  detailed  system 
design  decision.  This  "user"  may  be  performing  a  particular  analysis  with  known  data 
and  an  established  process,  or  pursuing  multiple  decision  alternatives  involving 
estimated  data  and  a  personalized  decision  methodology.  And,  equally  important,  the 
"user's"  level  of  knowledge  of  the  personnel,  the  tasks,  the  aiding/training 
environment,  and  the  JATAT  system,  as  well  as  other  similar  dimensions,  is  likely  to 
vary  within  each  analysis. 

The  number  of  individuals  employing  the  JATAT  system,  the  multitude  of 
purposes  for  which  the  system  is  employed,  and  the  variety  of  limits  and  abilities  of 
each  potential  user  requires  an  interface  with  the  power  and  flexibility  to  support  the 
needs  of  the  user  at  any  level,  at  any  given  step  in  the  process.  Good  human  factors 
principles  demand  that  the  complexity  of  the  design  be  transparent  to  the  user.  This 
notion  of  transparency  implies  that  by  the  nature  of  the  decisions  and  choices  the  user 
makes  at  each  juncture  in  the  system,  their  level  of  capability  is  inherently  defined. 
This  eliminates  the  need  for  explicit  user  classification  of  their  level  of  required 
assistance. 

In  order  to  identify  the  system  support  requirements  and  create  a  useful  interface 
metaphor  for  such  an  environment,  our  design  team  created  three  user  categories 
based  on  expected  capabilities;  novice,  journeyman,  and  expert.  Each  level  of 
expertise  category  is  defined  below. 

A  novice  JATAT  user  is  defined  as  having  fairly  limited  expertise  in  each  of  the 
following  areas;  the  domain  of  the  Air  Force  Specialty  (AFS)  under  analysis,  the 
aiding/training  problem  space,  modeling,  and  the  JATAT  software  package.  The 
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novice’s  interaction  with  the  system  is  expected  to  be  constrained  to  supplying 
available  input  values  (i.e.,  personnel,  equipment,  and  task  data,  design  changes, 
etc.),  explicitly  accepting  default  values  and  system  recommendations,  and  executing 
recommended  procedures.  Conceptually,  the  system  would  be  responsible  for 
guiding  data  input  through  methodological  queries,  and  providing  default  values  (i.e., 
decisions,  models,  etc.)  and  best  recommendations  based  on  available  taxonomies 
and  expert  heuristics. 

The  journeyman  is  also  expected  to  have  limited  expertise  in  the  AFS  domain  but 
is  more  likely  to  be  moderately  knowledgeable  regarding  the  aiding/training  problem 
space  and  modeling,  and  to  have  had  previous  experience  exercising  the  JATAT 
software  package.  The  interaction  of  the  journeyman  with  the  system  would  be  less 
constrained  and  more  proactive.  At  this  level  the  system  would  supply  the  user  with 
several  alternatives  for  mappings,  recommendations,  models/parameters,  etc.  along 
with  the  original  default  values.  Selection  of  a  particular  alternative  would  then  allow 
the  journeyman  more  control  in  guiding  the  direction  of  the  analysis.  An  important 
notion  at  this  level  is  that  the  system  remain  "knowledgeable"  about  each  decision 
made  by  the  user  (i.e.,  the  available  taxonomies  and  heuristics  are  sufficiently  robust 
to  map  to  the  potential  responses  of  the  user). 

The  expert  is  likely  to  be  well  versed  in  all  areas  of  the  analysis,  lacking  only 
specific  knowledge  regarding  the  particular  AFS  and  task  under  analysis.  This  level  of 
expertise  implies  the  ability  (and  possibly  the  desire)  to  deviate  from  standard  decision 
practices  in  pursuing  unique  solutions.  These  characteristics  define  the  highest  level 
of  flexibility  required  by  the  system  interface.  At  each  step  in  the  process,  the  system 
should  provide  the  appropriate  decision  template  and  recommendations/alternatives, 
while  allowing  completely  unconstrained  user  input.  This  level  of  support  portrays  the 
JATAT  methodology  concept  in  its  purest  form;  a  guided  aiding/training  decision 
framework. 

It  should  be  noted  that  the  system's  allowance  of  unconstrained  input  would  limit 
its  ability  to  provide  intelligent  guidance  and  recommendations  to  those  inputs  for 
which  taxonomies  and  expert  heuristics  exist  in  its  knowledge  bases.  At  a  minimum, 
the  system  should  identify  those  user  input  values  that  are  not  supported. 

Since  the  user's  level  of  expertise  can  vary  for  each  step  of  the  process  analysis,  a 
break-out  by  process  step  and  user  category  was  generated  to  assist  in  the 
identification  of  potential  system  support  functions.  (See  Table  2.) 
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Table  2.  Nature  of  JATAT  User  Support 


The  Environments 


While  investigating  alternative  means  for  simplifying  access  to  the  functionality  of 
the  JATAT  15  step  process,  it  was  discovered  that  many  of  the  system  support 
functions  were  inherently  useful  independent  of  the  methodology.  In  other  words, 
aggregating  numerous  modeling  tools  into  one  common  application  or  creating  a 
single  unified  interface  to  various  databases  results  in  an  inherent  "value  added"  to  the 
system  (i.e.,  the  power  of  the  tools  independent  of  the  process).  Further  pursuit  of  this 
idea  resulted  in  the  formulation  of  a  concept  of  independent  resources  centralized  in  a 
separate  system  component. 

In  response  to  this  discovery,  it  was  necessary  to  create  a  user/system  interface 
metaphor  which  would  intuitively  delineate  the  support  and  analysis  functions 
associated  with  the  15  step  methodology  from  those  generated  as  part  of  the  "value- 
added"  concept.  The  result  was  two  parallel  environments  living  within  the  same 
"world"  (system).  Between  these  two  environments  the  user  can  easily  travel  via  the 
environment  "gateway". 

The  first  environment  contains  each  of  the  15  steps  associated  with  the  conceptual 
aiding/training  trade-off  decision  methodology.  When  followed  sequentially,  these 
steps  describe  the  "process"  by  which  the  analyst  formulates  and  analyzes  the 
aiding/training  tradeoffs  for  a  given  design.  This  environment  is  referred  to  as  the 
"Process  Environment". 

The  "value-added"  concept  evolved  from  the  aggregation  of  several  inherent 
system  support  functions.  The  independent  utility  of  these  functions  served  as 
"resources"  for  the  user  in  a  variety  of  applications.  Hence,  the  second  interface 
environment,  an  aggregation  of  these  resources,  is  referred  to  as  the  "Resource 
Environment". 

The  Process  Environment 

The  purpose  of  the  Process  Environment  sub-section  is  to  describe,  in  detail,  each 
of  the  15  steps  of  the  JATAT  methodology  as  implemented  in  the  Phase  2  prototype 
system.  The  issues  addressed  by  each  of  the  following  15  segments  include  the 
user/system  interface  design,  user  input  requirements,  and  current  system  capabilities. 
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1 .  Understand  the  Job.  Initiating  a  successful  aiding/training  decision  analysis 
requires  an  initial  compilation  of  the  appropriate  knowledge,  including  the  decision 
catalysts  (i.e.,  those  situations  whose  consequential  impact  necessitates  a  new 
aiding/training  analysis;  for  example,  new/modified  equipment  designs,  or 
new/modified  manpower/personnel  requirements),  the  affected  factors  (i.e.,  personnel, 
equipment,  and  tasks),  and  candidate  solutions  (i.e.,  aiding  devices  and  training 
methods  which  satisfy  environmental  constraints)  pertinent  to  that  specific  situation. 

While  these  requirements  apply  to  all  aiding/training  analyses  regardless  of  the 
availability  of  computer-based  assistance,  its  importance  to  the  JATAT  user  is  due  to 
the  system's  reliance  upon  direct  user  input  of  certain  vital  data.  In  the  absence  of 
such  data,  subsequent  knowledge-based  heuristics  are  less  effective  and  the  resultant 
responses  less  accurate.  With  no  direct  means  of  inferring  the  knowledge  gained  by 
the  user  during  this  process,  the  computer-based  system  must  implicitly  ascertain  this 
information  through  user  responses  to  system  queries.  Information  of  this  type  can 
then  be  fed  directly  into  subsequent  analysis  processes  to  assist  in  making  obvious 
choices  among  alternatives  and  influence  performance  and  behavioral  predictions.  In 
essence,  this  knowledge  applies  to  both  the  user  and  the  computer-based  system;  the 
user  must  understand  the  job  in  order  for  the  system  to  do  the  job. 

During  the  development  of  the  JATAT  system  concept,  we  assumed  the  user  to 
have  some  level  of  knowledge  regarding  the  special  conditions  motivating  the 
analysis,  as  well  as  the  job  aiding/training  decision  domain  in  general.  While  this  is 
plausible,  it  is  less  likely  that  they  will  have  a  similar  level  of  knowledge  regarding  the 
personnel,  equipment,  and  tasks  being  analyzed.  Therefore,  some  method  by  which 
this  information  can  be  obtained  must  be  incorporated  into  a  fully  functional  support 
system.  While  the  integration  of  context  specific  data  (i.e.,  decision  catalysts  and 
environmental  constraints)  may  be  beyond  the  scope  of  the  JATAT  concept,  on-line 
access  to  databases  of  relevant  archival  data  is  well  within  the  purview  of  such  a 
computer-based  decision  support  system. 

JATAT  accessible  data  banks  would  most  reasonably  include  survey  data 
pertaining  to  the  operational  target  personnel,  equipment,  and  tasks  for  which 
aiding/training  analyses  would  be  performed.  In  this  case,  knowledge  of  the  target 
personnel  must  include  the  AF  specialty  and  related  job  responsibilities,  previous  job 
related  experience,  individual  aptitude  along  relevant  dimensions,  level  of  formal 
education,  current  training  procedures,  etc.  Knowledge  of  the  equipment  must  include 
some  level  of  understanding  of  the  technology  and  the  design  employed.  And, 
knowledge  of  the  tasks  requires  the  ability  to  identify  those  tasks  which  are  directly 
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affected  by  changes  to  the  job  responsibilities  and  equipment  as  specified  in  the 
analysis  scenario. 

Accessing  this  knowledge  across  a  variety  of  data  sources  would  best  be 
accomplished  through  a  single,  unified  interface.  For  example,  in  Figures  1  and  2,  the 
information  from  numerous  data  sources  (i.e. ,  the  Occupational  Research  Data  Bank 
(ORDB),  AF  Regulations,  Career  Development  Manuals,  etc.)  have  been  made 
available  through  a  common,  interactive  interface  method.  A  well-implemented 
support  system  would  maintain  this  common  interface  for  both  embedded  and  remote 
access  databases.  Due  to  storage  and  resource  limitations  the  current  DSS  prototype 
has  embedded  only  subsets  of  the  ORDB  data  for  the  AFS’s  being  analyzed  (i.e., 
426X2  and  911X0)  and  on-line  access  to  remote  data  banks  has  not  been 
implemented. 

Of  primary  importance  during  this  step  of  the  process  is  the  user  specification  of 
the  task  or  tasks  which  are  affected  by  the  decision  catalysts.  Specific  knowledge  of 
an  affected  task  is  highly  improbable  and  the  large  number  of  tasks  assigned  to  each 
AFS  diminishes  the  practicality  of  an  exhaustive  sequential  search.  Therefore,  some 
form  of  intelligent  search  through  an  existing  task  listing  is  required.  Currently,  a 
keyword  search  is  implemented  for  locating  the  task  of  interest.  Other  potential  search 
schemes  could  include  a  direct  numeric  access  procedure  or  a  scrolling  categorical 
search. 

In  the  absence  of  a  designated  AFS  or  equipment  about  which  to  reason,  the 
JATAT  methodological  framework  could  still  produce  reasonable,  although  much  less 
accurate,  recommendations.  But  an  ill-chosen  task  or  the  complete  absence  thereof 
results  in,  at  worst,  misleading  recommendations  or,  at  best,  no  recommendations  at 
all. 


Z _ Decompose  via  Task  Taxonomy.  Once  the  target  personnel  for  a  given 

aiding/training  analysis  have  been  determined,  the  associated  individual  and 
equipment  data  extracted  from  the  available  databases,  and  the  appropriate  affected 
task(s)  specified,  the  next  step  is  to  decompose  the  task(s)  to  the  level  most 
appropriate  for  the  analysis.  For  the  purpose  of  this  report,  a  single  level  of  task 
decomposition  results  in  a  number  of  task-related  activities  referred  to  here  as  sub¬ 
tasks  or  activities.  It  is  this  sub-task/activity  level  at  which  the  current  JATAT  system 
version  is  designed  to  perform.  This  decomposition  of  the  task(s)  can  be  based  upon 
either  pre-established  lists  of  sub-tasks/activities  or  general  taxonomies;  both  of  which 
can  be  easily  provided  by  the  system. 
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1. _ Understand  the  Job  a  42  6X2 
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REVIEW  OF  ARCHIVAL  DATA 

0RD8:  Occupational  Research  Data  Bank 
AF  Regulations:  39-10 
Equipment  Descriptions:  Technical  Manuals 
Training  Manuals:  CDCs 
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Figure  1.  Prototype  Archival  Data  Interface 


1.  Understand  the  Job 


OCCUPATIONAL  RESEARCH  DATA  BANK 


426X2 


You  have  accessed  the  Occupational  Research  Data  Bank 
at  the  AF  Human  Resources  Laboratory,  Brooks  AFB,  Texas. 
The  following  subsystems  are  currently  available  to  you. 

CARL  Reports:  Computer-Assisted  Reference  Locator 

ERIS  Data:  Enlisted  AFSC  Information  Subsystem 

Statistical  Data:  Distributions  of  Data  by  AFSC 

CODAP  Reports:  Selections  from  the  Occupational  Survey 
Data 
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Figure  2.  Prototype  ORDB  Interface 


The  pre-established  activities  are  formalized  procedures  extracted  from  current 
technical  manuals  and  are  inextricably  linked  to  specific  tasks.  Direct  database 
linkages  between  these  technical  activities  and  their  associated  tasks  could  be 
encoded  directly  into  the  JATAT  system,  such  that  selection  of  one  would  result  in  the 
automatic  retrieval  of  the  other.  This  is  the  most  accurate  method  of  task 
decomposition. 

A  series  of  general  activity  taxonomies  could  also  be  provided  for  those  classes  of 
tasks  not  having  pre-established  technical  procedures  (See  the  example  in  Figure  3). 
This  solution  enables  the  continuation  of  the  analysis  in  the  absence  of  specific 
activities.  However,  the  guidance  and  recommendations  produced  are  at  a 
significantly  lower  level  of  accuracy. 

Once  the  decomposition  has  occurred,  it  is  necessary  for  the  user  to  further  pare 
the  general  list  down  to  those  activities  specifically  impacted  by  the  decision  catalysts 
and  to  modify  them  to  appropriately  reflect  the  changes  in  the  target  system.  The 
capabilities  and  knowledge  necessary  to  perform  this  action  implicitly  define  the 
minimum  level  of  knowledge  of  the  user  necessary  to  successfully  complete  this  step 
of  the  analysis  process. 

At  this  point  in  the  current  framework,  the  user  is  expected  to  have  attained  a 
sufficient  understanding  of  the  personnel  and  job  parameters  to  employ  other 
available,  non-task  dependent  aiding/training  decision  models.  Interactive  versions  of 
the  Job  Performance  Aid  Selection  Algorithm  (Booher,  1978)  and  Training/Job 
Performance  Aid  Model  (Irvin,  Blunt  and  Lamb,  1988)  have  been  implemented  in  the 
Phase  2  DSS  prototype.  (See  Figure  4.) 

3a.Ass.ess  Limitations.  Abilities,  and  Preferences.  Determining  an  appropriate 
training  method  or  aiding  device  for  assisting  the  performance  of  a  particular  activity 
requires  an  understanding  of  the  two  following  components  of  that  activity;  the  activity 
itself  and  the  individual’s  ability  to  perform  that  activity.  The  former  was  resolved  in  the 
previous  step.  The  limits  and  abilities  of  the  target  personnel  are  the  focus  of  this  step. 

Individual  limits  and  abilities  can  be  defined  in  terms  of  knowledge  and  skills.  In 
the  current  system,  limits  are  operationally  defined  as  the  lack  of  knowledge  required 
to  perform  the  specified  activity.  Conversely,  abilities  are  implicitly  defined  as  an 
individual’s  capacity  to  manifest  a  minimum  level  of  knowledge  about  the  specified 
activity.  (Individual  skills  and  preferences  in  performing  the  selected  activities  were 
not  addressed  in  the  Phase  2  effort.) 
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Figure  3.  Example  Task  Decomposition  Taxonomy 
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The  knowledge  associated  with  a  given  activity  can  be  further  classified  as  either 
operational  or  system.  Operational  knowledge  refers  to  information  about  the  way  in 
which  tasks  are  performed  (i.e.,  "How  to  work  the  system").  System  knowledge  refers 
to  information  about  the  equipment  system  with  which  operators  and  maintainers 
perform  their  tasks  (i.e.,  "How  the  system  works")  (Rouse  and  Johnson,  1989).  (See 
Figure  5.) 

Within  each  knowledge  type  the  specific  knowledge  requirements  can  also  be 
categorized  into  subsets  designed  to  answer  the  questions  of  "how",  "what",  and  "why". 
Knowledge  requirements  can  also  be  ranked  upon  several  levels  of  abstractness,  from 
the  very  concrete  factual  information  to  the  abstract  principles  and  theories.  When 
categorized  along  both  dimensions  simultaneously,  nine  specific  knowledge 
requirements  emerge  for  both  operational  and  system  knowledge  classes.  (See 
Figures  6  and  7.) 

Although  the  terms  operational  and  syst^  '  .owledge  apply  most  directly  to  the 
operator  or  maintainer  as  the  target  personnel,  the  schema  remains  intact  for  other  job 
types  as  well.  The  structure  of  the  underlying  knowledge  bases  responsible  for 
mapping  limits  and  abilities  to  aiding/training  recommendations  also  remains 
consistent.  In  fact,  in  order  to  analyze  activities  as  varied  as  instructing  and 
troubleshooting  it  is  only  necessary  to  recompose  the  descriptive  language  of  each 
knowledge  requirement  relative  to  the  activity  being  analyzed.  For  example. 
Principles  of  Group  Interaction  is  more  applicable  to  the  instructional  task  of  a  911X0 
than  is  the  Mathematical  Principles/Theories  associated  with  the  maintenance  task  of 
a  426X2.  Note,  however,  that  both  denote  a  highly  abstract  level  of  operational 
knowledge  about  their  respective  domain.  (See  Figure  8.) 

It  is  conceivable  that  the  relevant  knowledge  requirements  would  be  pre- 
established  for  each  current  activity  and  would  be  automatically  recalled  with  each 
activity  analysis;  however,  it  will  still  be  necessary  for  the  JATAT  user  to  be  able  to 
specify  differences  of  knowledge  requirements  due  to  the  changes  of  interest.  This 
choice  identifies  the  minimum  level  of  ability  necessary  for  even  a  novice  user  at  this 
step  in  the  process. 

The  second  aspect  of  target  personnel  limits  and  abilities  addresses  the  issue  of 
overall  trainability.  Used  in  this  context,  trainability  predicts  those  knowledge 
requirements  that  can  not  be  satisfied  using  a  training  method  alone.  Although  not 
affected  by  the  specific  task  being  analyzed,  trainability  is  influenced  by  the 
technological  state  and  design  of  the  target  equipment.  (See  Figure  9.)  Currently,  the 
trainability  algorithm  also  requires  ordinal  measures  of  aptitude  and  experience. 
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Figure  5.  Assessing  Target  Personnel  Limits  and  Abilities 
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Figure  6.  Operational  Knowledge 
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Figure  7.  System  Knowledge 
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Figure  8.  Instructing  Knowledge 
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Figure  9.  Computing  Trainability 


4.  Map  Limitations,  etc,  to  Taxonomy  of  Training  Alternatives.  The  process  of 
mapping  the  previously  established  knowledge  requirements  to  a  viable  training 
recommendation  is  performed  automatically  at  the  system  level  based  on  the  expert 
rules  encoded  in  the  system  knowledge  base. 

Currently,  two  additional  parameters  also  influence  the  mapping  outcome; 
trainability  (as  established  in  the  previous  step)  and  effectiveness/efficiency  emphasis. 
The  latter,  although  implemented  within  the  previous  step,  acts  as  a  user  manipulated 
modifier.  This  is  a  preliminary  method  by  which  such  factors  as  training  cost  and 
availability,  time  allotted,  etc.  (i.e.,  knowledge  not  readily  available  to  the  system)  can 
indirectly  affect  the  recommendation  analysis.  Emphasizing  effectiveness  implies  a 
need  for  task  proficiency  regardless  of  cost.  Emphasizing  efficiency  implies  a  limited 
availability  of  resources  to  accomplish  the  training. 

Once  the  user  has  satisfactorily  responded  to  all  the  necessary  input  queries  for  a 
specified  activity,  the  system  will  respond  with  recommended  training  methods  for  both 
the  operational  and  system  knowledge  requirements  (See  Figure  10.).  Associated 
with  each  recommendation  is  a  "next  best"  alternative.  At  this  point,  the  JATAT  user 
can  continue  the  analysis  with  both  options,  or  proceed  after  choosing  to  continue  with 
only  the  recommendation  or  the  alternative. 

Note  that  in  Figure  10  two  recommendations/alternatives  are  given  for  both 
operational  and  system  knowledge  types.  This  is  due  to  the  expert  heuristics  which 
allow  the  possibility  of  a  different  type  of  training  method  for  the  "why"  knowledge 
versus  the  "how"  and  “what"  knowledge. 

A  list  of  the  specific  knowledge  requirements  satisfied  by  each 
recommended/alternative  training  method  is  also  provided.  Although  this  information 
is  most  useful  for  the  most  advanced  JATAT  users,  its  derivative,  those  knowledge 
requirements  that  remain  unsatisfied,  is  an  important  part  of  the  subsequent  "Make 
Obvious  Choices"  step. 

The  rules  implemented  in  the  Phase  2  DSS  prototype  for  deriving  recommended 
training  methods  were  based  on  the  taxonomies  identified  in  the  final  report  for  the 
Phase  1  effort. 

5.  Map  Limitations,  etc,  to  Taxonomy  of  Aiding  Alternatives.  As  in  the  preceding 
step,  aiding  device  recommendations/alternatives  are  automatically  derived  from 
previous  user  inputs.  In  fact,  the  aiding  device  recommendations  and  alternatives 
provided  by  the  Phase  2  prototype  system  are  based  on  a  subset  of  the  input  factors 
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associated  with  the  previous  training  recommendations/alternatives  (i.e.,  operational 
knowledge,  relative  importance  of  effectiveness/efficiency,  and  trainability). 

Although  the  current  aiding  recommendation  scheme  is  based  upon  the 
knowledge  requirements  associated  with  the  analyzed  activity,  it  is  hypothesized  that 
there  may  be  a  more  direct  link  between  the  activity  itself  and  available  aiding  devices. 
Exploration  of  this  potential  link  and  associated  heuristics,  as  well  as  other  candidate 
input  factors,  is  still  under  consideration. 

As  explicitly  noted  on  Figure  11.,  and  implicitly  acknowledged  in  the  list  of  satisfied 
knowledge  requirements,  system  knowledge  cannot  be  aided.  Preliminary 
investigation  indicates  that  only  a  combination  of  training  methods  and  design 
automation  will  suffice  in  this  case. 

6.  Make  Obvious  Choices.  The  purpose  of  this  step  is  to  identify  those 
aiding/training  recommendations/alternatives  which  satisfy  a  significant  portion  of  the 
aiding/training  requirements  independent  of  potential  combinations  with  other 
recommendations/alternatives.  This  process  involves  two  basic  tacts;  the  exploration 
of  untrainable  system  knowledge  requirements,  and  the  consideration  of  activity 
specific  factors. 

The  importance  of  untrainable  knowledge  requirements  stems  from  the  inability  of 
aiding  devices  to  serve  as  satisfactory  solutions  for  achieving  sufficient  levels  of 
proficiency  regarding  system  knowledge.  If  a  system  knowledge  requirement  is 
untrainable  (i.e.,  target  personnel  cannot  achieve  satisfactory  levels  of  knowledge  of 
the  specified  activity  through  available  training  methods)  the  "obvious  choice"  is  to 
automate  or  redesign  that  activity.  (See  Figure  12.)  This  solution  reveals  an 
underlying  JATAT  philosophy:  a  less  than  satisfactory  solution  to  a  given  knowledge 
requirement  is  always  considered  unacceptable. 

Activity  specific  factors  influence  the  "obvious  choice’  decision  by  considering 
ancillary  issues  inherent  to  the  activity  or  equipment  in  question.  For  example,  the 
existence  and  extent  of  Automatic  Test  Equipment  (ATE)  and  Built-in  Tests  (BITs) 
within  a  particular  system  will  greatly  influence  the  methods  by  which  a 
troubleshooting  task  is  trained  or  aided.  Continuation  of  the  JATAT  analysis  into  the 
modeling/simulation  phase  is  based  on  the  number  and  characteristics  of  the 
remaining  aiding/training  recommendations/alternatives  after  the  paring  process  of 
this  step. 
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7.  Coalesce  Tradeoffs  Involving  Interdependent  Tasks.  The  by-product  of 
removing  the  "obvious  choice"  aiding/training  recommendations/alternatives  from  the 
compilation  of  all  possible  solutions  in  the  preceding  step  is  a  complimentary  list  of 
aiding  devices  and  training  methods  for  which  no  unique  advantages  exist.  At  this 
level  of  the  analysis,  one  or  more  of  the  aiding/training  alternatives  appear  to  satisfy 
the  specified  knowledge  requirements  with  some  level  of  parity.  This  ambiguity 
provides  the  basis  for  the  formulation  of  particular  aiding/training  tradeoffs  to  be 
analyzed  in  the  subsequent  modeling  phase  of  the  methodology. 

The  actual  trade-off  analysis  can  be  simplified  and  the  power  of  the  results 
increased  by  employing  an  intelligent  aggregation  scheme  across  related  activities 
and  subsequently  optimizing  the  trade-off  components.  The  current  aggregation 
philosophy  is  to  first  combine  candidate  aiding/training  solutions  according  to  the 
interdependency  of  certain  characteristics  of  the  activities.  For  example,  it  would  be 
reasonable  to  combine  all  troubleshooting  tasks  for  electronic  components  into  one 
group  and  those  associated  with  mechanical  components  into  another.  These  newly 
formed  groups  are  then  re-evaluated  for  further  aggregation  potential  based  on 
characteristics  of  the  aiding/training  alternatives.  For  instance,  within  the  electronic 
trouble-shooting  tasks  it  is  plausible  that  the  combined  on-the-job  training  and 
procedural  aid  solution  would  be  more  effective  than  either  alternative  alone.  (See 
Figure  13.)  It  is  these  solution  sets  which  are  traded  off  against  one  another  during  the 
modeling/simulation  phase  of  the  analysis.  (Note  that  this  coalescing  process  is 
"scripted"  in  the  Phase  2  prototype  system;  no  dynamic  functionality  has  been 
encoded.) 

An  interesting  side-effect  of  these  aggregations  is  the  potential  for  initiating 
another  iteration  of  the  "obvious  choice"  process.  Specific  activity-type  factors  will 
influence  the  evaluation  of  each  aiding/training  solution  set  much  as  the  individual 
aiding/training  solution  evaluations  were  influenced  during  the  first  execution  of  "Make 
Obvious  Choices".  If  the  outcome  of  this  process  is  only  one  viable  solution  set,  the 
analysis  is  considered  completed  and  the  modeling  portion  of  the  framework  is 
ignored.  Should  more  than  one  component  of  the  trade-off  be  accepted,  the 
preparatory  steps  of  applying  a  quantitative  model  are  initiated. 

8.  Choose  Measures  of  Performance.  Choosing  the  measures  of  performance  is 
the  first  in  a  series  of  steps  for  identifying  the  most  useful  quantitative  model  for 
analyzing  a  particular  trade-off.  The  performance  measure  selected  provides  the 
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dimension  across  which  each  trade-off  component  is  evaluated  relative  to  the  others. 
Example  measures  of  performance  include  mean  time  to  repair  (MTTR),  probability  of 
error,  number  of  errors,  number  of  components  (i.e.,  system  complexity),  etc.  (See 
Figure  14.) 

The  system’s  response  to  a  user's  selection  of  a  particular  measure  of 
performance  is  to  identify  the  available  quantitative  model(s)  which  produce  the 
appropriate  output  value. 

While  the  system  may  be  able  to  provide  some  descriptive  guidance  regarding  the 
measures  of  performance  it  has  knowledge  about,  it  is  still  the  responsibility  of  the  user 
to  determine  which  is  most  appropriate  for  the  intended  analysis.  This  situation 
defines  the  minimum  knowledge  necessary  for  the  novice  user  to  complete  this  step 
and  is  common  to  each  of  the  following  three  steps. 

9.  Choose  Input,  w  , .put  Representations.  The  second  step  in  the  process  of 
selecting  a  useful  qr  .ntitative  model  is  to  choose  the  input/output  representation  most 
appropriate  to  ihe  trade-off  being  analyzed.  ’Appropriate’,  in  this  case,  can  be 
addressed  as  either  of  two  dimensions;  that  which  most  accurately  represents  the  task, 
and  tha4  for  which  the  most  data  is  available.  Successful  execution  of  the  final 
quantitative  evaluation  (i.e.,  reliable,  accurate  output  values)  will  be  significantly 
affected  by  the  user's  ability  to  provide  the  model  with  an  accurate  description  of  the 
target  system. 

Comprehension  of  the  input/output  representation  is  an  important  factor  in  this 
step;  the  user  must  be  able  to  apply  the  chosen  input/output  representation  as  part  of 
the  execution  of  the  quantitative  model.  This  application  task  can  vary  from  altering 
single  parametric  values  for  individual  system  components,  to  providing  the  entire 
system  design  architecture,  to  a  generic  simulation  tool.  The  Phase  2  prototype 
system  permits  system  representation  through  either,  a  functional  or  a  physical 
perspective.  (See  Figure  15.) 

10.  Identify  Requisite  Structure  and  Parameters.  In  order  to  fully  support  the 
input/output  representation,  the  user  must  identify  the  structure  and  parameters  of  the 
quantitative  model.  The  purpose  of  this  structure  and  parameters  is  to  reflect  the 
machine  and  human  components  within  the  system.  Modeling  the  machine 
components  requires  certain  structures  such  as  timed  sequences  of  specified  discrete 
events  or  simulations  of  continuous  interactive  systems.  Modeling  the  human 
component  generally  requires  the  simulation  of  certain  procedures  or  strategies.  For 
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Figure  14.  Measures  of  Performance 
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Figure  15.  Input/Output  Representations 
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instance,  one  trouble-shooting  model  may  employ  a  "halt-split"  strategy  (i.e.,  the  goal 
of  each  test  is  to  reduce  the  search  space  for  the  failed  component  by  half}  as  its  action 
selection  criteria  while  another  may  employ  a  "least  cost  first"  strategy  (i.e.,  tests  are 
prioritized  to  minimize  incurred  costs).  In  this  case,  the  user  must  select  the  paradigm 
which  most  closely  resembles  the  action  selection  criteria  employed  by  the  target 
personnel  in  the  operational  environment. 

The  extent  of  this  step  will  be  defined  by  the  scope  of  the  input/output 
representation  and  the  availability  of  existing  quantitative  models.  Representations  of 
individual  components  or  representations  that  are  already  extensively  modeled  will 
only  require  the  identification  of  the  parameters  to  be  manipulated.  Other  situations 
may  require  extensive  analysis  to  identify  and  develop  the  underlying  structures  to 
support  the  quantitative  models. 

1 1.  If  Necessary.  Represent  Learning  Process.  The  execution  of  some  quantitative 
task  performance  models  requires  a  representation  of  the  learning  process  by  which 
the  target  personnel  achieve  proficiency  at  that  task.  As  mentioned  earlier  in  this 
report,  the  representation  of  these  learning  processes  can  range  from  simple  database 
retrievals  to  complex  learning  curve  models,  and  address  such  issues  as  cost,  time, 
retention,  etc.  The  user  is  responsible  for  selecting  the  most  accurate  representation 
for  the  specified  task  for  which  sufficient  data  is  available. 

12.  Apply  Methods  of  Analysis  to  Representations.  Based  on  user  selections  in 
the  four  preceding  steps,  the  JATAT  system  will  present  a  set  of  recommended  models 
(i.e.,  one  or  more  based  on  availability  and  selection  criteria)  applicable  to  the  trade-off 
being  analyzed.  Presentation  of  these  recommendations,  in  future  systems,  could  be 
in  two  forms;  a  hierarchical  structure  based  on  some  weighting  scheme  (i.e.,  number 
of  criteria  satisfied),  or  a  tabular  form  showing  the  entire  model  subset  and  associated 
dimensions. 

The  user  must  then  select  from  this  set  that  model  which  best  suits  his/her  needs 
and  apply  it  to  each  component  of  the  trade-off  in  question.  (See  Figure  16.)  The 
need  to  fully  execute  the  selected  model  for  each  component  of  the  trade-off  in 
question  implies  several  iterations  of  model  execution,  each  of  which  requires  unique 
user  supplied  input  parameter  values.  Managing  these  multiple  analysis  iterations 
requires  an  interface  design  which  clearly  indicates  the  trade-off  component  being 
evaluated  and  its  associated  parametric  values.  (See  Figure  17.) 
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12.  Apply  Methods  of  Analysis 


The  following  models  are  applicable  to 
the  tradeoff  analysis  (shown  below)  based 
upon  your  reponses  in  the  preceding  steps. 
Select  a  model  for  the  analysis. 


Models 


Maintainability  Prediction  Model 
Wohl  (1982) 

Fuzzy  Rute-8asod  Model 

Hunt  4  Rouse  (1984) 


TRADEOFF  #1 

Select  the  training/aiding  combination 
for  analysis. 


on  the  job  responsibility  full  scope  simulator 

and  and 

rocedural  aid  deductive  aid 
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Figure  16.  Applying  the  Methods  of  Analysis 


12.  Apply  Methods  of  Analysis 
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MAINTAINABILITY  PREDICTION  MODEL 
Wohl  (1982) 

Diagnostic  Event  Value 

0|  .85  10 

Complexity  Index 

01  5.0  to 

Average  Component  Test  Time 

01  0.5  O 

The  Mean  Time  To  Repair  (MTTR)  Is: 


3.305722 


on  the  job  responsibility 
and 

rocedural  aid 


Click  to  return  to  begin  neit 
analysis. 
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Figure  17.  Interface  for  Maintainability  Prediction  Model 
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The  model's  output  for  each  trade-off  component  analysis  is  a  numeric  value;  not  a 
decision.  The  final  decision  must  still  be  made  by  the  user  in  conjunction  with  the 
results  from  each  of  the  other  component  analyses.  This  decision  is  made  as  part  of 
the  "Interpret  Results"  step. 

13.  Interpret  Results.  Result  interpretation  has  two  fundamental  components;  the 
system  process,  and  the  user  decision. 

The  system  process  is  similarly  decomposed  into  two  components;  aggregation  of 
results  and  relative  assessments.  Aggregation  of  results  is  the  process  of  compiling 
the  output  values  from  each  execution  of  the  selected  model  and  presenting  them, 
along  with  their  associated  trade-off  components  and  input  parameter  values,  in  some 
comprehensive  manner.  This  step  is  necessary  due  to  the  potential  number  of  trade¬ 
off  components  and  the  variable  time  factor  for  a  given  analysis.  The  system  can  then 
perform  a  relative  assessment  of  the  output  values  along  some  appropriate  dimension 
(i.e.,  percent  difference,  ranking,  etc.)  to  assist  the  user’s  final  aiding/training  selection 
decision. 

As  mentioned  in  the  previous  step,  the  quantitative  modeling  process  results  in 
output  values  and  relative  assessments,  not  a  decision.  The  user  is  still  responsible 
for  making  the  final  decision.  This  decision  is  purely  textual  input  in  nature  and  is  not 
processed  or  reasoned  about  by  the  system.  The  decision  is  simply  integrated  into  the 
documentation  file  for  later  presentation. 

At  this  point  the  analysis  phase  of  the  process  is  complete,  some  level  of 
resolution  of  the  initial  problem  has  been  achieved,  and  the  remaining  steps  in  the 
methodology  are  to  simply  provide  documentation  functions  to  support  the 
presentation  and  defense  of  the  decision. 

14.  Compile  Assumptions  and  Consequences  of  Tradeoffs.  Due  to  the  volume  of 
information  being  generated  and  exchanged  between  the  system  and  user  during  a 
JATAT  analysis,  the  potential  for  long  periods  of  time  to  transpire  between  the 
performance  of  successive  steps  in  the  process,  and  the  need  to  document  the 
intermediate  decisions  leading  to  a  final  aiding/training  solution,  some  form  of 
automated  audit  trail  is  necessary  in  a  complex  decision  support  system. 

The  design  solution  as  envisioned  for  the  JATAT  DSS  prototype  is  a  log  file 
containing  each  user  input,  decision  made,  path  taken  and  system  response.  This  file 
is  maintained  in  real-time  during  each  analysis.  The  existence  of  such  a  log  file  will 


33 


allow  users  to  automatically  update  and  re-initialize  an  analysis  after  any  length  of 
delay  in  the  process,  as  well  as  output  the  stored  information  in  a  specified  report 
format  to  assist  the  user’s  defense  of  the  final  recommendation. 

15.  Form  Sets  of  Tradeoffs  with  Consistent  Assumptions  and. Consequences.  This 
final  step  is  included  to  aid  the  user  when  a  large  number  of  tradeoffs  have  been 
analyzed  as  a  result  of  a  single  originating  design  catalyst.  The  goal  is  to  assist  in 
identifying  commonalties  across  each  of  the  tradeoffs  permitting  a  more  efficient 
application  of  the  aiding/training  solution  sets  to  the  operational  environment. 

This  step  was  not  developed  for  the  Phase  2  DSS  prototype. 


The  Resource  Environment 

The  JATAT  support  functions  which  led  to  the  evolution  of  the  Resource 
Environment  have  been  categorized  into  the  following  five  resources;  Clerical,  Data 
Archives,  Modeling,  Process  Control,  and  Shipping/Handling.  The  following 
descriptions  are  provided  for  each  resource. 

Clerical.  The  objective  of  the  Clerical  resource  is  to  enable  system  log  file 
management  independent  of  accessing  the  15  steps  and  to  provide  tutorial  guidance 
regarding  system  editorial  capabilities.  Management  of  the  log  files  should  include 
aspects  of  directory  manipulation,  individual  file  parameter  and  commonality  searches, 
file  editing,  and  identification  of  individual  file  requirements. 

Data  Archives.  The  underlying  concept  of  the  Data  Archives  resource  is  to  enable 
access  to  various  existing  databases,  for  the  purpose  of  data  retrieval,  while 
employing  a  single  consistent  interface.  In  the  resource  environment  the  user  should 
be  allowed  access  to  the  information  across  any  dimension  (i.e.,  AFS,  task,  equipment, 
etc.)  and  should  not  be  restricted  to  those  databases  which  are  contained  within  the 
JATAT  system.  It  is  conceivable  that  communications  protocols  and  access 
parameters  would  be  incorporated  into  the  system  allowing  remote  access  to 
numerous  databases  that  are  useful  but  too  large  to  be  contained  within  the  JATAT 
system.  The  information  contained  within  the  Data  Archives  resource  should  also 
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include  mapping  and  decomposition  taxonomies,  databases  of  survey  data  and  expert 
heuristics,  and  on-line  versions  of  training  and  technical  manuals. 

Modeling.  The  Modeling  resource  is  designed  to  be  an  aggregation  of 
independent  quantitative  modeling  tools  for  solving  a  variety  of  computational 
problems.  Within  this  resource  the  user  should  be  able  to  investigate  and  exercise 
each  available  model  --  independent  of  a  complete  methodological  analysis. 
Modeling  resource  support  functions  should  include  guidance  regarding  associated 
measures  of  performance,  input/output  representations,  and  learning  models  as  well 
as  model  execution  and  result  interpretation. 

Process  Control.  The  Process  Control  resource  is  expected  to  function  as  a 
JATAT  methodology  tutorial.  The  function  and  role  of  each  process  step  would  be 
described  in  detail  with  a  sample  analysis  employed  as  an  example.  An  environment 
'gateway'  would  allow  direct  access  to  a  particular  step  in  the  process  environment 
from  its  corresponding  step  in  the  Process  Control  resource;  and  vice  versa. 

Shioping/Handling.  The  Manpower,  Personnel,  and  Training  (MPT)  domain 
space  encompasses  a  vast  number  of  issues  from  basic  training  graduate  requirement 
predictions  to  aircraft  sortie  rate  estimation.  JATAT  is  just  one  of  several  decision 
aiding  systems  and  simulation  models  in  development  or  currently  operational  (e.g., 
LCOM.SUMMA,  and  TDS),  designed  to  provide  accurate  solutions  to  these  needs.  In 
a  properly  integrated  environment,  data  generated  by  one  tool  would  be  used  as  input 
to  another. 

The  responsibility  of  the  Shipping/Handling  resource  would  be  to  assist  in  forming 
useful  links  (both  data  and  conceptual)  between  JATAT  and  other  known  models.  This 
objective  would  be  supported  through  a  tutorial  of  roles  and  requirements  of  other 
known  MPT  tools  and  a  data  processing  capability  which  would  format  the  JATAT 
results  and  decisions  to  expedite  the  potential  data  transfer  process. 

The  Advisor 

In  order  to  successfully  employ  an  aiding/training  decision  support  system  with  the 
level  of  complexity  equivalent  to  that  expected  of  a  fully  developed  JATAT  system,  the 
user  will  require  some  form  of  simple,  elegant,  yet  powerful  method  of  assistance. 
From  this  predicted  need  has  evolved  the  current  concept  of  the  intelligent  advisor. 
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The  role  to  be  played  by  this  intelligent  advisor  is  as  the  unobtrusive,  ubiquitous 
mentor.  This  implies  that  the  advisor  would  be  discreetly  aware  of  each  interaction  the 
user  has  with  the  system,  but  becomes  visible  for  advice  only  upon  command. 

The  advice  and  assistance  provided  must  span  several  levels  of  detail  and 
abstractness.  This  would  be  manifest  in  the  ability  to  identify  the  user’s  current  status 
in  the  overall  analysis  process,  recall  previous  inputs  and  decisions,  and  predict  likely 
implications  of  the  historical  behaviors.  The  advisor's  knowledge  repertoire  should 
also  include  a  history  of  the  user’s  movement  through  the  system,  decisions  made  at 
each  juncture,  data  input  and  data  deficits,  and  procedure  transgressions.  It  is  also 
important  that  the  advisor  be  able  to  provide  advice  at  various  levels  of  detail 
corresponding  to  each  category  of  user  abilities,  as  well  as  to  know  which  level  of 
advice  is  appropriate. 

The  current  implementation  of  the  advisor  function  is  "scripted"  and  very  context 
specific.  The  target  goal  for  such  a  function,  however,  would  likely  be  a  context- 
sensitive  expert  system. 


Supporting  Data  Types 

Similar  to  other  decision  aiding  paradigms,  the  models  necessary  to  support  the 
JATAT  methodology  are  extremely  information  dependent.  The  type/format  of  data 
required  by  each  component  of  the  decision  process  will  vary  as  a  function  of  the 
characteristics  of  the  intermediate  decision  constructs.  For  JATAT,  these  data 
requirements  can  be  classified  into  three  types;  archival  data,  user  provided  contextual 
information,  and  expert  rules/algorithms  inherent  to  the  system. 

Archival  data  is  operationally  defined,  for  the  purpose  of  this  report,  as  the  factual 
personnel,  task  and  equipment  information  typically  collected  through  occupational 
surveys  or  compiled  from  system  operation  and  maintenance  documents.  This 
includes  such  data  as  task/sub-task  listings,  performance  times/probabilities/and 
requirements,  task  and  equipment  complexity,  aptitude/experience,  and  training 
indicators.  In  the  JATAT  aiding/training  decision  support  paradigm,  archival  data 
serve  as  both  the  target  data  to  be  processed  by  the  JATAT  models,  and  as  the  user’s 
source  of  domain  knowledge.  Currently,  this  information  resides  in  a  miscellany  of 
data  sources  which  include  the  Occupational  Research  Data  Bank  (ORDB),  AF 
Regulations,  career  development  training  manuals,  Air  Training  Command  Plans  of 
Instruction  (POI’s),  and  equipment  technical  manuals. 
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Contextual  information  is  the  abstract,  highly  situation  specific  data  which 
represents  the  background  scenario  of  the  archival  data.  This  information  is  primarily 
characterized  by  its  relational  nature.  It  encompasses  those  factors  which  preceded 
(and  subsequently,  effected)  the  aiding/training  analysis  (i.e. ,  decision  catalysts),  and 
the  environmental  ramifications  (i.e.,  political,  temporal,  and  resource-related)  of  the 
candidate  aidingAraining  solutions. 

The  context  specificAemporal  nature  of  this  data  precludes  simple  codification. 
Rather,  a  JATAT  analysis  is  dependent  upon  the  user  as  the  sole  source  of  this 
information.  This  data  is  presented  to  the  system  in  two  manners;  direct  input  to 
system  queries  (e.g.,  choosing  to  emphasize  the  effectiveness  or  efficiency  of  the 
training  method)  or  indirect  process  guidance  decisions  (e.g.,  opting  to  analyze  only 
two  of  three  recommended  trade-off  components). 

System  inherent  data  contains  the  knowledge  which  guides  the  analysis  process, 
intelligently  responds  to  user  decisions,  provides  advice,  simulates  target  system 
performance,  and  makes  aidingAra'ning  recommendations.  This  data  is  primarily  in 
the  form  of  taxonomies,  and  expert  rules  resident  in  the  system  knowledge  bases. 
Included  in  this  category  are  potential  training/aiding  methods,  training  philosophies, 
operational/system  knowledge  requirements,  modeling  packages,  input/output 
parameters,  etc.  This  knowledge  is  established  during  the  system  design  and 
development  phases  and  is  obtained  through  knowledge  acquisition  sessions  with 
relevant  domain  experts. 


IV.  SUMMARY 

This  final  section  of  the  report  is  a  compilation  of  the  "lessons  learned"  during  the 
Phase  2  JATAT  concept  development,  execution,  and  implementation.  These 
"lessons",  which  address  such  issues  as  the  characteristics  of  the  data  required  to 
execute  the  methodology,  additional  support  functionality,  and  MPT  model 
integration/standardization,  can  provide  valuable  insight  to  future  aidingAraining 
decision  support  efforts. 

For  the  purpose  of  this  summary,  lessons  learned  have  been  divided  into  the 
following  two  categories;  those  addressing  the  development  of  an  implementable 
aiding/training  decision  support  methodology,  and  those  addressing  system 
application  oriented  functionality. 
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The  importance  of  formulating  universally  accepted,  standardized  task  definitions 
(i.e.,  terminology  and  level  of  resolution)  represents  a  single,  cogent  conclusion  drawn 
from  three  independent  issues.  First,  the  accuracy  and  responsiveness  of  a  computer- 
based  decision  aiding  application  is  directly  dependent  upon  the  mapping  of 
unfamiliar  task  to  well-understood  taxonomies.  Complicating  this  mapping  by 
representing  tasks  at  various  levels  of  abstraction  decreases  the  power  and  success  of 
the  aiding/training  recommendations.  The  second  issues  supporting  the  pursuit  of  a 
standardized  task  definition  effort  is  that  current  task  listings  lack  consistency  in  their 
granularity  of  task  specification.  For  example,  the  Air  Force  Occupational  Research 
DataBank  specifies  both  form  completion  and  propulsion  system  trouble-shooting  at 
the  task  level.  No  explicit  standard  is  established  regarding  time  required,  number 
and/or  type  of  activities  involved,  focus  of  aiding/training  solutions,  etc.  This 
inconsistency  among  critical  task  dimensions  also  complicates  the  process  of 
transitioning  information  among  different  decision  models.  Recent  Air  Force  emphasis 
on  MPT  model  integration,  the  third  issue,  is  directed  toward  developing  consistency 
and  relatedness  among  the  numerous  decision  models.  Definition  standardization 
within,  and  across,  models  and  databases  is  essential  to  facilitate  such  a  concept. 

Data  Availability 

In  the  pursuit  of  the  archival  data  necessary  to  perform  the  operational  AFS 
analyses  (see  Appendix  A)  for  this  effort,  it  became  evident  that  data  availability  is  an 
issue  of  great  importance  as  well.  Specifically,  data  availability  in  terms  of  the  data’s 
format  (i.e.,  how  closely  the  existing  data  format  conforms  to  that  required  by  the 
system  defines  the  degree  of  pre-processing  necessary)  and  the  medium  within  which 
the  data  resides  (i.e.,  hard,  bound  copies  vs.  a  computer-based  data  bank).  Each 
level  of  extrapolation,  interpolation  and  media  transformation  required  to  apply  the 
data  to  the  methodological  framework,  in  essence,  decreases  the  inherent  utility  of  the 
data.  Unfortunately,  current  data  sources  vary  widely  along  both  of  these  dimensions. 
Therefore,  commitment  and  effort  will  be  required  to  structure  available  data  sources  to 
more  directly  support  the  integrated  environment  of  the  many  new  decision  aiding  and 
modeling  systems. 


Knowledge  Deficits 

In  the  current  JATAT  framework,  Operational  and  System  Knowledge 
Requirements  are  specified  for  a  given  activity  independent  of  consideration  of  the 
personnel  responsible  for  performing  that  activity.  Hence,  the  system  generated 
aiding/training  recommendations  implicitly  assume  100%  knowledge  deficit  for  each 
specified  knowledge  requirement.  This  assumption  disregards  the  likelihood  of 
transfer  of  knowledge  based  previous  experience  and  training.  Continued 
investigation  is  necessary  to  identify  the  impact  of  varying  levels  of  knowledge  deficit 
on  the  formulation  of  aiding/training  recommendations. 

Individual  Skills 

This  phase  of  effort  focused  on  knowledge  as  the  dimension  along  which  the 
limitations  and  abilities  of  the  target  personnel  was  measured.  While  this  perspective 
provided  the  foundation  for  aiding/training  recommendation  heuristics,  it  was  also 
clear  that  individual  skill  plays  an  equally  important  role  in  this  selection  process. 
Future  development  of  these  aiding/training  rules  must  consider  the  impact  of  skill 
related  limitations  and  abilities  and  their  interaction  with  previously  established 
knowledge-based  heuristics. 

Application  Oriented  Functionality 

Simulation  Environment 

The  availability  of  simulation  models  for  predicting  human-system  performance  is 
highly  domain  dependent.  For  instance,  while  there  are  a  fair  number  of  maintenance 
and/or  operations  models  in  existence,  the  availability  of  similar  models  for 
instructional  or  clerical  tasks  is  extremely  limited,  if  at  all  existent.  Frequently,  those 
models  or  tools  which  do  exist  are  so  highly  specific  that  the  analysis  of  related,  but 
slightly  different  tasks  are  currently  unsupported. 

To  create  and  implement  models  or  tools  to  support  every  potential  task  analysis 
would  be  an  insurmountable  effort.  More  reasonably,  this  situation  necessitates  a 
single,  robust  simulation  environment  which  can  be  applied  generally  to  a  class  of 
tasks. 

Modeling  Tutorial 
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Another  related  functionality  essential  to  an  operational  JATAT  system  is  a 
modeling-domain  tutorial.  One  assumption  made  regarding  each  potential  JATAT 
user  was  that  while  they  would  be  fairly  knowledgeable  about  the  aiding/training 
domain  in  general,  they  would  be  less  so  regarding  the  particular  the  application  of 
specific  performance  prediction  models.  In  this  case,  it  is  imperative  that  a  final  JATAT 
system  provide  some  level  of  tutorial  functionality  for  accessing  and  executing  each  of 
the  available  quantitative  models.  This  would  include  start-up,  data  input,  and  result 
interpretation  instructions  for  those  models  developed  independently  of  the  JATAT 
system,  as  well  as  the  aforementioned  generic  simulation  environment. 

Model  Selection 

In  the  current  methodological  scheme,  quantitative  model  selection  is  based  on  a 
series  of  sequential  steps  during  which  model  characteristics  applicable  to  the  specific 
analysis  are  identified.  However,  the  performance  of  the  operational  AFS  analyses 
(as  described  in  Appendix  A)  indicated  that  this  process  need  not  be  so  highly 
sequential.  It  appears,  instead,  that  this  process  can  occur  with  equal  success  given 
the  same  selections  regardless  of  order.  As  a  result,  future  interface  functionality 
development  should  be  directed  at  supporting  this  phenomenon. 

In  summary,  the  success  of  the  Phase  2,  Job  Aiding/Training  Allocation 
Technologies  effort  has  been  threefold.  It  has  resulted  in  the  development  of  a 
general  aiding/training  trade-off  decision  methodology,  the  application  of  this 
methodology  to  two  independent  analyses  of  operational  AFS's,  and  the 
implementation  of  this  methodology  into  a  computer-based  decision  support 
demonstration  system.  While  these  successes  make  no  implications  regarding  the 
maturity  and  readiness  of  the  technology,  they  do  indicate  that  the  solution  is  tenable. 
It  will  be  further  research  into  the  areas  of  the  input  factor  relationships  within 
aiding/training  domain,  continued  trade-off  decision  methodology  formulation, 
supporting  database  development,  predictive  model  generation,  and  the  design  of 
decision  support  system  functionality  that  will  help  bring  the  JATAT  concept  to  reality. 
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APPENDIX  A 


OPERATIONAL  AFS  ANALYSES 

The  purpose  of  this  section  is  to  document  the  analyses  of  the  two  AFHRL 
approved  AFS's  (i.e.,  426X2  and  911X0)  using  the  decision  framework  and  expert 
heuristics  developed  during  this  phase  of  the  JATAT  effort.  The  dialogue  of  inputs, 
decisions,  and  recommendations  between  the  user  and  the  system  is  documented  for 
each  step  in  the  JATAT  methodology  to  assist  in  guiding  the  user  through  each 
complete,  DSS  aided  analysis  process. 


.426X2*.  Jet_EaflmeZIur.bo-Pr.op.  Maintenance-Analysis 


Scenario: 

The  AF  has  recently  approved  the  modification  of  the  Pratt-Whitney  F-100  Engine. 
One  component  of  this  modification  includes  replacing  the  original  electro-mechanical 
fuel  flow  controller  with  a  newly  developed  solid-state  controller.  Maintenance  of  this 
engine  and  the  respective  fuel  system  is  the  responsibility  of  the  426X2,  Jet 
Engine/Turbo-Prop  Maintenance  Technician.  The  task  of  the  JATAT  user  is  to  identify 
the  necessary  changes  to  the  current  training/aiding  scheme  for  fuel  flow  controller 
maintenance. 


1  JJrafersland  the  Job 

The  user  initiates  a  directed  search  for  the  426X2  AFSC  and  the  system  responds 
with  the  available  archival  data  for: 

AFS:  426X2,  Jet  Engine/Turbo-Prop  Maintenance  Technician 

The  user's  knowledge  of  this  AFS  is  sparse  so  a  review  of  the  available  data 
including  AF  Reg  39-10,  the  CDC  Manuals  and  the  ORDB  reports  and  statistics  is 
conducted. 
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2.  Decompose  via  Task  Taxonomy 

During  the  ORDB  data  review,  the  user  accesses  the  list  of  tasks  for  which  the 
426X2  is  responsible.  The  large  number  of  identified  tasks  (i.e.,  approximately  700) 
precludes  an  efficient  sequential  search  for  those  tasks  affected  by  the  design  change. 
Therefore,  the  task  search  space  is  narrowed  through  an  automated  search  on  the  key 
word  "fuel".  The  user  selects  only  one  of  the  returned  tasks  for  analysis: 

Task:  G322  Isolate  Malfunction  with  Engine  Fuel  System 

(It  is  the  only  one  which  deals  directly  with  the  engine  fuel  flow  controller.) 

Decomposition  of  this  task  occurs  automatically  based  upon  pre-established 
procedural  activities  as  defined  in  the  equipment  technical  maintenance  manuals. 

Activities: 

G322  A  Verify  power  to  the  fuel  pump 
G322  B  Perform  fuel  flow  controller  diagnostics 
G322  C  Inspect  fuel  injector  nozzles 
G322  D  Test  fuel  divider 

The  user  then  identifies  activities  B  and  D  as  those  affected  by  the  proposed 
design  change  and  edits  them  accordingly. 

Edited  Activities: 

G322  B  Perform  electronic  fuel  flow  controller  diagnostics 
G322  D  Test  fuel  divider  solenoid 


3.  Assess  Limitations.  Abilities,  and  Preferences 

The  user  identifies  the  knowledge  requirements  (limits)  associated  with  each 
target  task  from  the  Operational  and  System  Knowledge  taxonomies. 

G322  B  Perform  electronic  fuel  flow  controller  diagnostics 
Knowledge  Requirement  Limits  (Operational) 
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Situations 

Procedures 

Operational  Basis 

Criteria 

Strategies 

Logical  Basis 

Methodologies 

Knowledge  Requirement  Limits  (System) 
Characteristics  of  System  Elements 
Functioning  of  System  Elements 
Requirements  Fulfilled 
Relationships  Among  System  Elements 
Co-Functioning  of  System  Elements 
Temporal  Patterns  of  System  Response 
Overall  Mechanism  of  System  Response 


G322  D  Test  fuel  divider  solenoid 

Knowledge  Requirement  Limits  (Operational) 
Situations 
Procedures 
Criteria 

Knowledge  Requirement  Limits  (System) 
Characteristics  of  System  Elements 
Functioning  of  System  Elements 
Relationships  Among  System  Elements 
Co-Functioning  of  System  Elements 
Temporal  Patterns  of  System  Response 


4.  Map  Limitations.  Etc,  to  Taxonomy  of  Training  Alternatives 

The  user  then  specifies  the  relative  importance  of  effectiveness  and  efficiency  for 
determining  the  training  recommendations  and  alternatives. 
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Effectiveness  =  High 
Efficiency  =  Low 

A  final  factor  the  system  requires  prior  to  formulating  the  training  and  aiding 
recommendations  is  a  general  level  of  ability  (relative  to  the  knowledge  requirements) 
the  target  personnel  can  be  expected  to  achieve.  This  is  represented  as  the 
trainability  of  the  personnel  and  is  computed  based  upon  user  input  to  the  following 
queries. 

Aptitude:  Medium 
Experience:  High 
Category  of  Technology:  New 
Category  of  Design:  New 

The  system  determines  a  trainability  level  of  three  (3)  for  an  individual  in  this 
situation. 

Based  on  this  input  and  the  knowledge  requirements  chosen  in  the  previous  step, 
the  system  responds  with  the  following  training  recommendations  and  alternatives  and 
presents  the  knowledge  requirements  satisfied  by  that  particular 
recommendation/alternative. 


G322  B  Perform  electronic  fuel  flow  controller  diagnostics 
Active  Training  =  Full-Scope  Simulator 

On-the-job  Responsibility  (Alternate) 

Situations 
Procedures 
Operational  Basis 
Criteria 
Strategies 

Passive  Training  =  Classroom  Instruction 

Classroom  Discussion  (Alternate) 
Characteristics  of  System  Elements 
Functioning  of  System  Elements 
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Requirements  Fulfilled 
Relationships  Among  System  Elements 
Co-Functioning  o*  System  Elements 
Temporal  Patterns  of  System  Response 


G322  D  Test  fuel  divider  solenoid 

Active  Training  =  Full-Scope  Simulator 

On-the-job  Responsibility  (Alternate) 

Situations 

Procedures 

Criteria 

Passive  Training  =  Classroom  Instruction 

Classroom  Discussion  (Alternate) 
Characteristics  of  System  Elements 
Functioning  of  System  Elements 
Relationships  Among  System  Elements 
Co-Functioning  of  System  Elements 
Temporal  Patterns  of  System  Response 


5.  Mao  Limitations.  Etc,  to  Taxonomy  of  Aiding  Alternatives 

No  further  input  is  required  of  the  user  for  this  step;  the  recommendations  are 
based  on  previous  inputs.  Simple  user  access  of  this  step  is  all  that  is  required  to 
investigate  the  system’s  aiding  recommendations.  The  DSS  indicates  that  no  System 
Knowledge  Requirements  are  satisfied  by  the  prescribed  aiding 
recommendations/alternatives  while  noting  that  they  cannot  be  satisfied  through  the 
use  of  aiding  devices.  (This  situation  is  dealt  with  during  the  Make  Obvious  Choices 
step.) 


G322  B  Perform  electronic  fuel  flow  controller  diagnostics 
Aiding  Device  =  Procedural  Aid 

Deductive  Aid  (Alternate) 

Situations 
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Procedures 
Operational  Basis 
Criteria 
Strategies 

G322  D  Test  fuel  divider  solenoid 
Aiding  Device  =  Procedural  Aid 

Deductive  Aid  (Alternate) 

Situations 

Procedures 

Criteria 


6.  Make  Obvious  Choices 

The  level  of  resolution  provided  by  the  DSS  mapping  heuristics  allows  this  step  to 
be  performed  automatically  by  the  system.  Thus,  the  user  must  simply  access  the  step 
in  order  to  investigate  the  system’s  recommendations.  Since  aiding  devices  are  not 
applicable  for  the  System  Knowledge  Requirements,  Obvious  Choice 
recommendations  are  initially  based  on  those  which  can  be  trained  and  indicates 
those  knowledge  requirements  which,  based  on  the  pre-established  trainability  level, 
can  not  be  trained. 

G322  B  Perform  electronic  fuel  flow  controller  diagnostics 

Training  Method  Recommendation: 

Classroom  Instruction 

Untrainable  System  Knowledge  Requirements: 

Overall  Mechanism  of  System  Response 

(Requires  automation  or  system  redesign  to  satisfy.) 

G322  D  Test  fuel  divider  solenoid 

Training  Method  Recommendation: 

Classroom  Instruction 

Untrainable  System  Knowledge  Requirements: 

None 
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7.  .Coalesce  Tradeoffs  Involving  Interdependent  Tasks 

At  this  point  in  the  process,  the  aiding/training  recommendations  and  alternatives 
are  combined  across  activities  to  form  viable  sets  of  aidingAraining  solutions.  Each  set 
forms  one  component  of  the  overall  trade-off  decision. 


Component  #1: 
On-the-job  Responsibility 
with 

Procedural  Aid 


Component  #2: 
Full-scope  Simulator 
with 

Deductive  Aid 


The  user  may  accept  this  aggregation  and  continue  with  the  analysis  or  reject  it 
and  formulate  their  own.  Acceptance  ensures  continued  system  support. 

The  system  may  also  enter  an  iterative  loop  at  this  point  (based  upon  known 
characteristics  of  the  decision  domain  and  the  trade-off  in  question)  to  establish  more 
Obvious  Choice  selections.  This  would  be  indicated  by  specialized  DSS  queries. 
User  responses  to  these  queries  would  result  in  different  paths  through  the  remaining 
analysis.  For  example: 

Are  Automatic  Test  Equipment  (ATE)  or  Built-In  Tests  (BIT)  available? 

If  ATE  and  BIT  are  not  available,  only  component  #2  is  applicable;  the  trade-off 
decision  is  non-existent  and  the  analysis  could  be  considered  complete.  If  ATE  and 
BIT  are  available,  both  components  remain  applicable  and  the  analysis  must  be 
performed  to  delineate  them.  (User  responds  to  the  affirmative  to  this  query.) 


^  Choose  Measures  of  Performance 

Currently  only  four  measures  of  performance  are  available  for  selection  (mean 
time  to  repair,  probability  of  error,  number  of  errors,  and  number  of  components 
changed);  conceivably  many  more  could  be  made  available.  Selection  of  a  particular 
measure  of  performance  results  in  a  system  response  which  indicates  those  models 
which  satisfy  that  performance  measure  requirement. 
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The  user's  choice  of  Mean  Time  to  Repair  (MTTR)  is  answered  with  a  system 
recommendation  to  employ  the  Maintainability  Prediction  Model  (Wohl,  1982). 


9.  Choose  Input-Output  Representations 

The  input/output  representation  indicates  the  perspective  with  which  the 
quantitative  model  views  the  problem  space.  The  two  available  representations  are 
physical  (e.g.,  a  schematic  diagram)  and  functional. 

The  user  opts  to  use  a  functional  representation  for  this  analysis.  The  available 
models  which  employ  this  form  of  representation  include  the  Maintainability  Prediction 
Model  (Wohl,  1982)  and  the  Fuzzy  Rule-Based  Model  (Hunt  &  Rouse,  1984). 


IQ.  Identify  Requisite  Structures  and  Parameters 

These  structures  and  parameters  are  decision  rules  inherent  to  the  model 
algorithm. 

The  user  knows  that  a  "half-split"  test  node  selection  strategy  is  used  by  the 
maintenance  technicians  in  the  field.  This  particular  troubleshooting  strategy  is 
considered  an  Action  Selection  Criterion  parameter  and  is  thus  selected.  A  Task 
Analysis  is  Not  Available  for  this  scenario.  The  resultant  model  is,  again,  the 
Maintainability  Prediction  Model  (Wohl, 1982). 


jJL  If  Necessary.  Represent  Learning  Process 

These  Learning  Representations  usually  include  some  form  of  learning  curves,  or 
human  memory  model  applicable  to  that  particular  task  and  model. 

This  step  is  not  applicable  to  the  models  under  consideration. 


.12,  Apply  Methods  of  Analysis  to  Representations 

The  user  must  select  the  Method  of  Analysis  (model)  to  execute  for  the  analysis  of 
this  particular  trade-off. 

The  Maintainability  Prediction  Model  (Wohl, 1982)  is  chosen.  Each  individual 
component  of  the  trade-off  is  then  analyzed  separately  using  the  same  model.  The 
following  values  are  supplied  to  the  model. 
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Component  #1: 

On-the-job  Responsibility 
with 

Procedural  Aid 

Diagnostic  Event  Value  =  .85 
Complexity  Index  =  5.0 
Avg.  Comp.  Test  Time  =  .5 


Component  #2: 

Full-scope  Simulator 
with 

Deductive  Aid 

Diagnostic  Event  Value  =  .85 
Complexity  Index  =  2.5 
Avg.  Comp.  Test  Time  =  .5 


1 3.  Interpret  Results 

The  system  compiles  all  component  analyses  and  presents  them  in  one  cohesive 
report  with  a  low  level  analysis  of  the  relative  values. 

Component  #1 :  Component  #2: 

On-the-job  Responsibility  Full-scope  Simulator 

with  with 

Procedural  Aid  Deductive  Aid 

MTTR  =  3.306  MTTR  =  1.153 


The  MTTR  for  component  #1  is  approximately  100%  greater  than  for  component 

#2. 


14,  Compile  Assumptions  and  Consequences  of  Tradeoffs 

The  system  has  retained  all  inputs,  decisions,  recommendations/alternatives, 
computational  output  as  part  of  the  log  file.  The  user  may  chose  to  save  or  delete  this 
file  and/or  print  a  hard  copy. 


15.  Form  .Sets  of  Tradeoffs  with  Consistent  Assumptions  and  Consequences 

Since  only  one  trade-off  was  analyzed,  this  step  is  declared  Not  Applicable  by  the 
system. 
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911X0,  Aerospace  Physiology  Analysis 


Scenario: 

With  recent  discoveries  of  the  effects  of  high  altitude  flight  on  aircrew  members  the 
Air  Force  has  instituted  a  new  instructional  requirement  for  all  aircrews  regarding 
aircraft  pressurization.  The  course  will  be  a  major  revision  of  the  current  aircraft 
pressurization  course  taught  by  AF  medical  technicians,  911X0’s,  as  an  extension  to 
the  current  in-flight  physiology  program.  The  course  has  been  designed  to  include  the 
latest  in  instructional  aids  and  techniques  so  a  training/aiding  profile  has  been 
requested  to  assist  these  technicians  in  presenting  the  new  material. 


1 .  Understand  the  Job 

The  user  initiates  a  directed  search  for  the  911X0,  Aerospace  Physiology  AFSC 
and,  based  on  personal  experience  with  previous  analyses  of  this  specialty,  traverses 
the  system  directly  to  the  Decompose  via  Task  Taxonomy  step. 


2.  Decompose  via  Task  Taxonomy 

The  user  accesses  the  Occupational  Survey  Research  (OSR)  list  of  tasks 
established  for  this  AFS,  narrows  the  search  space  through  an  automated  search  on 
the  key  phrase,  and  inputs  "conduct  classroom  instruction".  The  following  task  is 
selected  from  those  which  satisfied  the  search  phrase. 


Task:  G199  Conduct  Classroom  Instruction  on  Principles  of  Aircraft 
Pressurization 

Due  to  the  variety  of  instructional  tasks  available,  decomposition  of  this  task  is 
based  on  the  generic  activity  taxonomy  associated  with  conducting  classroom 
instruction. 


Activities: 
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G199  A  Knowledge  about  Subject  Matter 

G19$  B  Knowledge  about  Instruction 

G199C  Presentation 

G199D  Interaction 

G199E  Assessment 

G199  F  Diagnosis 

G199G  Remediation 

The  user  identifies  all  activities  as  being  affected  by  the  proposed  new  course  and 
assumes  them  to  be  sufficient  for  the  current  analysis. 


3.  Assess  Limitations.  Abilities,  and  Preferences 

The  user  identifies  the  knowledge  requirements  (limits)  associated  with  each 
target  task  from  the  Operational  and  System  Knowledge  taxonomies. 


G199  A  Knowledge  about  Subject  Matter 
Assume  No  Deficits 


G199  B  Knowledge  about  Instruction 

Knowledge  Requirement  Limits  (Instructing) 

Episodes  and  Situations  Which  Can  Arise  in  the  Instructional 
Setting 

Procedures  for  Dealing  with  Episodes  and  Situations  in  the 
Instructional  Setting 
Instructional  Priorities 

Knowledge  Requirement  Limits  (Instructional) 

Characteristics  of  Elements  of  Instructional  System 


G199C  Presentation 

Knowledge  Requirement  Limits  (Instructing) 
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Episodes  and  Situations  Which  Can  Arise  in  the  Instructional 
Setting 

Procedures  for  Dealing  with  Episodes  and  Situations  in  the 
Instructional  Setting 
Instructional  Priorities 

Knowledge  Requirement  Limits  (Instructional) 

Characteristics  of  Elements  of  Instructional  System 


G199D  Interaction 

Knowledge  Requirement  Limits  (Instructing) 

Episodes  and  Situations  Which  Can  Arise  in  the  Instructional 
Setting 

Procedures  for  Dealing  with  Episodes  and  Situations  in  the 
Instructional  Setting 
Instructional  Priorities 

Strategies  for  Dealing  with  Episodes  and  Situations  in  the 
Instructional  Setting 

Knowledge  Requirement  Limits  (Instructional) 

Characteristics  of  Elements  of  Instructional  System 
Functioning  of  Elements  of  Instructional  System 
Relationships  Among  Elements  of  Instructional  System 


G199E  Assessment 

Knowledge  Requirement  Limits  (Instructing) 

Episodes  and  Situations  Which  Can  Arise  in  the  Instructional 
Setting 

Procedures  for  Dealing  with  Episodes  and  Situations  in  the 
Instructional  Setting 
Instructional  Priorities 

Knowledge  Requirement  Limits  (Instructional) 

Characteristics  of  Elements  of  Instructional  System 
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G199F  Diagnosis 

Knowledge  Requirement  Limits  (Instructing) 

Episodes  and  Situations  Which  Can  Arise  in  the  Instructional 
Setting 

Procedures  for  Dealing  with  Episodes  and  Situations  in  the 
Instructional  Setting 
Instructional  Priorities 

Knowledge  Requirement  Limits  (Instructional) 

Characteristics  of  Elements  of  Instructional  System 


G199G  Remediation 

Knowledge  Requirement  Limits  (Instructing) 

Episodes  and  Situations  Which  Can  Arise  in  the  Instructional 
Setting 

Procedures  for  Dealing  with  Episodes  and  Situations  in  the 
Instructional  Setting 
Instructional  Priorities 

Knowledge  Requirement  Limits  (Instructional) 

Characteristics  of  Elements  of  Instructional  System 
Functioning  of  Elements  of  Instructional  System 
Relationships  Among  Elements  of  Instructional  System 


iu  Mao  Limitations.  Etc,  to  Taxonomy  of  Training  Alternatives 

The  user  then  specifies  the  relative  importance  of  effectiveness  and  efficiency  for 
determining  the  training  recommendations  and  alternatives. 

Effectiveness  =  Low 
Efficiency  =  High 


54 


A  final  factor  the  system  requires  prior  to  formulating  the  training  and  aiding 
recommendations  is  a  general  level  of  ability  (relative  to  the  knowledge  requirements) 
the  target  personnel  can  be  expected  to  achieve.  This  is  represented  as  the 
trainability  of  the  personnel  and  is  computed  based  upon  user  input  to  the  following 
queries. 

Aptitude:  Medium 
Experience:  Low 
Category  of  Technology:  Old 
Category  of  Design:  New 

The  system  determines  a  trainability  level  of  two  (2)  for  an  individual  in  this 
situation. 

Based  on  this  input  and  the  knowledge  requirements  chosen  in  the  previous  step, 
the  system  responds  with  the  following  training  recommendations  and  alternatives  and 
presents  the  knowledge  requirements  satisfied  by  that  particular 
recommendation/alternative. 


G199A  Knowledge  about  Subject  Matter 
G199B  Knowledge  about  Instruction 
Active  Training  =  Student  Teaching 

On-the-job  Responsibility  (Alternate) 

Episodes  and  Situations  Which  Can  Arise  in  the  Instructional 
Setting 

Procedures  for  Dealing  with  Episodes  and  Situations  in  the 
Instructional  Setting 
Instructional  Priorities 
Passive  Training  =  Classroom  Instruction 

Classroom  Discussion  (Alternate) 

Characteristics  of  Elements  of  Instructional  System 


G199C  Presentation 

Active  Training  =  Student  Teaching 
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On-the-job  Responsibility  (Alternate) 

Episodes  and  Situations  Which  Can  Arise  in  the  Instructional 
Setting 

Procedures  for  Dealing  with  Episodes  and  Situations  in  the 
Instructional  Setting 
Instructional  Priorities 
Passive  Training  =  Classroom  Instruction 

Classroom  Discussion  (Alternate) 

Characteristics  of  Elements  of  Instructional  System 


G199D  Interaction 

Active  Training  =  Student  Teaching 

On-the-job  Responsibility  (Alternate) 

Episodes  and  Situations  Which  Can  Arise  in  the  Instructional 
Setting 

Procedures  for  Dealing  with  Episodes  and  Situations  in  the 
Instructional  Setting 
Instructional  Priorities 

Strategies  for  Dealing  with  Episodes  and  Situations  in  the 
Instructional  Setting 

Passive  Training  =  Classroom  Instruction 

Classroom  Discussion  (Alternate) 

Characteristics  of  Elements  of  Instructional  System 
Functioning  of  Elements  of  Instructional  System 
Relationships  Among  Elements  of  Instructional  System 


G199E  Assessment 

Active  Training  =  Student  Teaching 

On-the-job  Responsibility  (Alternate) 

Episodes  and  Situations  Which  Can  Arise  in  the  Instructional 
Setting 

Procedures  for  Dealing  with  Episodes  and  Situations  in  the 
Instructional  Setting 
Instructional  Priorities 
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Passive  Training  =  Classroom  Instruction 

Classroom  Discussion  (Alternate) 
Characteristics  of  Elements  of  Instructional  System 


G199F  Diagnosis 

Active  Training  =  Student  Teaching 

On-the-job  Responsibility  (Alternate) 

Episodes  and  Situations  Which  Can  Arise  in  the  Instructional 
Setting 

Procedures  for  Dealing  with  Episodes  and  Situations  in  the 
Instructional  Setting 
Instructional  Priorities 
Passive  Training  =  Classroom  Instruction 

Classroom  Discussion  (Alternate) 

Characteristics  of  Elements  of  Instructional  System 


G199G  Remediation 

Active  Training  =  Student  Teaching 

On-the-job  Responsibility  (Alternate) 

Episodes  and  Situations  Which  Can  Arise  in  the  Instructional 
Setting 

Procedures  for  Dealing  with  Episodes  and  Situations  in  the 
Instructional  Setting 
Instructional  Priorities 
Passive  Training  =  Classroom  Instruction 

Classroom  Discussion  (Alternate) 

Characteristics  of  Elements  of  Instructional  System 
Functioning  of  Elements  of  Instructional  System 
Relationships  Among  Elements  of  Instructional  System 


5.  Map  Limitations.  Etc,  to  Taxonomy  of  Aiding  Alternatives 

No  further  input  is  required  of  the  user  for  this  step;  the  recommendations  are 
bases  on  previous  inputs.  Simple  user  access  of  this  step  is  all  that  is  required  to 
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investigate  the  system's  aiding  recommendations.  The  DSS  indicates  that  no  System 
Knowledge  Requirements  are  satisfied  by  the  prescribed  aiding 
recommendations/alternatives  while  noting  that  they  cannot  be  satisfied  through  the 
use  of  aiding  devices.  (This  situation  is  dealt  with  during  the  Make  Obvious  Choices 
step.) 


G199A  Knowledge  about  Subject  Matter 
Aiding  Device  =  Computer-Based  Instruction 

Procedural  Check-list  (Alternate) 

Episodes  and  Situations  Which  Can  Arise  in  the  Instructional 
Setting 

Procedures  for  Dealing  with  Episodes  and  Situations  in  the 
Instructional  Setting 
Instructional  Priorities 


G199  B  Knowledge  about  Instruction 

Aiding  Device  =  Computer-Based  Instruction 

Procedural  Check-list  (Alternate) 

Episodes  and  Situations  Which  Can  Arise  in  the  Instructional 
Setting 

Procedures  for  Dealing  with  Episodes  and  Situations  in  the 
Instructional  Setting 
Instructional  Priorities 


G199C  Presentation 

Aiding  Device  =  Computer-Based  Instruction 

Procedural  Check-list  (Alternate) 

Episodes  and  Situations  Which  Can  Arise  in  the  Instructional 
Setting 

Procedures  for  Dealing  with  Episodes  and  Situations  in  the 
Instructional  Setting 
Instructional  Priorities 
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G199D  Interaction 

Aiding  Device  =  Computer-Based  Instruction 

Procedural  Check-list  (Alternate) 

Episodes  and  Situations  Which  Can  Arise  in  the  Instructional 
Setting 

Procedures  for  Dealing  with  Episodes  and  Situations  in  the 
Instructional  Setting 
Instructional  Priorities 

Strategies  for  Dealing  with  Episodes  and  Situations  in  the 
Instructional  Setting 


G199E  Assessment 

Aiding  Device  =  Computer-Based  Instruction 

Procedural  Check-list  (Alternate) 

Episodes  and  Situations  Which  Can  Arise  in  the  Instructional 
Setting 

Procedures  for  Dealing  with  Episodes  and  Situations  in  the 
Instructional  Setting 
Instructional  Priorities 


G199F  Diagnosis 

Aiding  Device  =  Computer-Based  Instruction 

Procedural  Check-list  (Alternate) 

Episodes  and  Situations  Which  Can  Arise  in  the  Instructional 
Setting 

Procedures  for  Dealing  with  Episodes  and  Situations  in  the 
Instructional  Setting 
Instructional  Priorities 


G199G  Remediation 

Aiding  Device  =  Computer-Based  Instruction 

Procedural  Check-list  (Alternate) 
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Episodes  and  Situations  Which  Can  Arise  in  the  Instructional 
Setting 

Procedures  for  Dealing  with  Episodes  and  Situations  in  the 
Instructional  Setting 
Instructional  Priorities 


6.  Make  Obvious  Choices 

The  level  of  resolution  currently  provided  by  the  DSS  mapping  heuristics  allows 
this  step  to  be  performed  automatically  by  the  system.  Thus,  the  user  must  simply 
access  the  step  in  order  to  investigate  the  system's  recommendations.  Since  aiding 
devices  are  not  applicable  for  the  System  Knowledge  Requirements,  Obvious  Choice 
recommendations  are  initially  based  on  those  which  can  be  trained  and  indicates 
those  knowledge  requirements  which,  based  on  the  pre-established  trainability  level, 
can  not  be  trained. 

All  Activities  have  same  recommendation: 

G322  A  Knowledge  about  Subject  Matter 

G322  B  Knowledge  about  Instruction 

G322  C  Presentation 

G322  D  Interaction 

G322  E  Assessment 

G322  F  Diagnosis 

G322  G  Remediation 

Training  Method  Recommendation: 

Classroom  Instruction 

Untrainable  System  Knowledge  Requirements: 

None 

User  Enters: 

Class  Size:  Large 
Course  Frequency:  Moderate 
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The  user  then  responds  (negative)  to  a  query  requesting  the  availability  of 
Computer-Based  Instruction.  The  system  responds  by  formulating  a  single 
aiding/training  component:  Student  Teaching  with  Procedural  Checklist. 

With  only  one  viable  aiding/training  component,  no  tradeoffs  exist  and,  therefore, 
further  analysis  is  not  applicable.  The  user  can  directly  go  to  the  Compile 
Assumptions  step. 


1 -^Compile  Assumptions  and  Consequences  of  Tradeoffs 

The  system  has  retained  all  inputs,  decisions,  recommendations/alternatives, 
computational  output  as  part  of  the  log  file.  The  user  may  chose  to  save  or  delete  this 
file  and/or  print  a  hard  copy. 


1IL  Form  Sets  of  Tradeoffs  with  Consistent  Assumptions  and  Consequences 
Not  Applicable. 


% 
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